00000100:00100000:3.0:1789760691.410877:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493617920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.469622:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972312 gen 3 00000008:00010000:3.0:1789760691.469686:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760691.469695:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.469716:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.469738:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760691.470907:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493617920:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760691.470967:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:3.0:1789760691.471087:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 20455424<>20459520 00000100:00100000:3.0:1789760691.471139:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493618048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.525464:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972313 gen 3 00000008:00010000:3.0:1789760691.525515:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760691.525523:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.525537:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.525557:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760691.525586:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493618048:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760691.591975:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760691.591993:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.592006:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.592098:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760691.592130:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493618176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.594254:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760691.594285:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.645590:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972315 gen 3 00000008:00010000:3.0:1789760691.645676:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760691.645689:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.645701:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.645732:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760691.645777:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.648817:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760691.756463:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972316 gen 3 00000008:00010000:3.0:1789760691.756515:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760691.756524:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760691.756536:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760691.756559:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760691.756591:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618432:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760691.984690:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5000, 5000] 00000080:00200000:3.0:1789760691.984717:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760691.984724:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760691.984740:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20484096 i_blocks 40967 00000020:00200000:3.0:1789760691.984744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760691.984755:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760691.984791:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20484096, count: 4096 00000080:00200000:3.0:1789760691.984799:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20484096, bytes: 4096 00000080:00200000:3.0:1789760691.984808:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760691.984824:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760691.984826:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20484096, 20488192) 0 0 00000080:00200000:3.0:1789760691.984831:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760691.984847:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5001, 5001] 00020000:00200000:3.0:1789760691.984855:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20484096 chunk: [20484096, 20488192) 20488192, 4096 00020000:00200000:3.0:1789760691.984860:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760691.984880:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5001, 5001] 00000008:00010000:3.0:1789760691.984898:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760691.984905:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760691.984918:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20484096, 20488192) append 0 append lockpos 0 00000080:00200000:3.0:1789760691.984932:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20484096=0x1389000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760691.984993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760691.984996:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20488192 00000020:00010000:3.0:1789760691.985004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760691.985007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.985010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760691.985013:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760691.985016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760691.985030:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760691.985037:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760691.985053:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760691.985154:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 20484096<>20488192 00000100:00100000:3.0:1789760691.985210:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760691.985228:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760692.037555:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972319 gen 3 00000100:00100000:3.0:1789760692.037600:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149880 x1876699493618944/t4294972320(4294972320) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760707 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760692.037652:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760692.037660:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.037670:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.037696:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760692.037731:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493618944:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760692.747042:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5010, 5010] 00000080:00200000:3.0:1789760692.747075:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760692.747083:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760692.747103:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20525056 i_blocks 40967 00000020:00200000:3.0:1789760692.747109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760692.747126:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760692.747161:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20525056, count: 4096 00000080:00200000:3.0:1789760692.747169:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20525056, bytes: 4096 00000080:00200000:3.0:1789760692.747183:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760692.747191:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760692.747194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20525056, 20529152) 0 0 00000080:00200000:3.0:1789760692.747198:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760692.747214:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5011, 5011] 00020000:00200000:3.0:1789760692.747223:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20525056 chunk: [20525056, 20529152) 20529152, 4096 00020000:00200000:3.0:1789760692.747228:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760692.747251:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5011, 5011] 00000008:00010000:3.0:1789760692.747277:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760692.747284:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760692.747296:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20525056, 20529152) append 0 append lockpos 0 00000080:00200000:3.0:1789760692.747311:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20525056=0x1393000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760692.747379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760692.747382:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20529152 00000020:00010000:3.0:1789760692.747389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760692.747437:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.747440:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.747444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760692.747447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760692.747456:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760692.747462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760692.747484:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760692.747604:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 20525056<>20529152 00000008:00010000:3.0:1789760692.836540:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760692.836559:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.836572:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.836598:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760692.836630:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493620224:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760692.937876:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5012, 5012] 00000080:00200000:3.0:1789760692.937892:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760692.937898:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760692.937912:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20533248 i_blocks 40967 00000020:00200000:3.0:1789760692.937916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760692.937925:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760692.937957:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20533248, count: 4096 00000080:00200000:3.0:1789760692.937964:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20533248, bytes: 4096 00000080:00200000:3.0:1789760692.937973:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760692.937982:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760692.937985:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20533248, 20537344) 0 0 00000080:00200000:3.0:1789760692.937989:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760692.938005:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5013, 5013] 00020000:00200000:3.0:1789760692.938023:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20533248 chunk: [20533248, 20537344) 20537344, 4096 00020000:00200000:3.0:1789760692.938029:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760692.938049:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5013, 5013] 00000008:00010000:3.0:1789760692.938066:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760692.938072:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760692.938084:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20533248, 20537344) append 0 append lockpos 0 00000080:00200000:3.0:1789760692.938097:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20533248=0x1395000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760692.938152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760692.938155:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20537344 00000020:00010000:3.0:1789760692.938164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760692.938168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.938172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760692.938176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760692.938179:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760692.938187:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760692.938194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760692.938210:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760692.938308:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 20533248<>20537344 00000100:00100000:3.0:1789760692.938351:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760692.938368:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760692.989083:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972332 gen 3 00000008:00010000:3.0:1789760692.989153:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760692.989161:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760692.989168:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760692.989235:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760692.989259:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620480:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760693.020027:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5014, 5014] 00000080:00200000:3.0:1789760693.020045:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760693.020052:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760693.020067:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20541440 i_blocks 40967 00000020:00200000:3.0:1789760693.020071:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760693.020081:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760693.020115:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20541440, count: 4096 00000080:00200000:3.0:1789760693.020122:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20541440, bytes: 4096 00000080:00200000:3.0:1789760693.020133:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760693.020142:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760693.020144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20541440, 20545536) 0 0 00000080:00200000:3.0:1789760693.020149:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760693.020167:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5015, 5015] 00020000:00200000:3.0:1789760693.020177:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20541440 chunk: [20541440, 20545536) 20545536, 4096 00020000:00200000:3.0:1789760693.020182:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760693.020219:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5015, 5015] 00000008:00010000:3.0:1789760693.020239:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760693.020246:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760693.020257:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20541440, 20545536) append 0 append lockpos 0 00000080:00200000:3.0:1789760693.020274:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20541440=0x1397000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760693.020332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760693.020335:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20545536 00000020:00010000:3.0:1789760693.020343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760693.020348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.020351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760693.020354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760693.020357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760693.020364:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760693.020371:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760693.020388:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760693.020528:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 20541440<>20545536 00000100:00100000:3.0:1789760693.020578:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.060866:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972334 gen 3 00000008:00010000:3.0:1789760693.060918:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760693.060927:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.060940:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.060994:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760693.061025:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.061689:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493620864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.116811:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972335 gen 3 00000008:00010000:3.0:1789760693.116884:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760693.116893:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.116907:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.116936:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760693.116965:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493620864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.118063:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.195857:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972336 gen 3 00000008:00010000:3.0:1789760693.195906:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760693.195914:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.195927:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.195946:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760693.195976:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493620992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.197136:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760693.197157:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493621120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.274236:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972337 gen 3 00000008:00010000:3.0:1789760693.274288:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760693.274297:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.274309:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.274570:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760693.274601:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493621120:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760693.484727:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760693.619963:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493621888:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.687828:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972343 gen 3 00000008:00010000:3.0:1789760693.687900:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760693.687909:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.687920:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.687941:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760693.687968:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493621888:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.688759:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760693.688778:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.754973:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972344 gen 3 00000008:00010000:3.0:1789760693.755025:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760693.755034:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.755047:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.755067:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760693.755098:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.756039:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.815659:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972345 gen 3 00000008:00010000:3.0:1789760693.815724:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760693.815732:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.815755:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.815773:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760693.815803:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.816682:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760693.816716:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.877930:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972346 gen 3 00000008:00010000:3.0:1789760693.877984:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760693.877992:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.878004:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.878072:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760693.878100:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.879067:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.931199:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972347 gen 3 00000008:00010000:3.0:1789760693.931245:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760693.931253:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.931264:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.931281:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760693.931306:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.932027:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760693.932043:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622528:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760693.970625:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760693.970640:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760693.970652:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760693.970673:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760693.970713:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760693.972017:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.024918:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972349 gen 3 00000008:00010000:3.0:1789760694.024984:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760694.024994:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.025005:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.025025:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760694.025053:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.025842:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.025859:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.065915:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972350 gen 3 00000008:00010000:3.0:1789760694.065968:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760694.065977:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.065988:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.066009:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760694.066039:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.066817:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.137875:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972351 gen 3 00000008:00010000:3.0:1789760694.137927:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760694.137935:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.137946:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.137967:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760694.137997:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493622912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.139284:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.139306:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493623040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.198580:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972352 gen 3 00000100:00100000:3.0:1789760694.379709:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493623680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.430692:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972357 gen 3 00000008:00010000:3.0:1789760694.430751:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760694.430760:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.430773:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.430792:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760694.430822:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493623680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.431807:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.431827:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493623808:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760694.517833:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760694.517851:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.517863:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.517885:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760694.517914:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493623808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.630966:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972359 gen 3 00000008:00010000:3.0:1789760694.631032:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760694.631041:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.631054:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.631085:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760694.631115:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493623936:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.633972:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.634006:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.726003:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972360 gen 3 00000008:00010000:3.0:1789760694.726053:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760694.726062:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.726086:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.726105:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760694.726134:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.727696:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.819469:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972361 gen 3 00000008:00010000:3.0:1789760694.819523:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760694.819532:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.819546:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.819566:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760694.819596:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.820722:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.820751:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.887124:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972362 gen 3 00000008:00010000:3.0:1789760694.887170:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760694.887177:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.887189:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.887209:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760694.887240:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.889104:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.954998:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972363 gen 3 00000008:00010000:3.0:1789760694.955050:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760694.955059:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760694.955070:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760694.955091:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760694.955120:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760694.956093:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760694.956111:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.012515:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972364 gen 3 00000008:00010000:3.0:1789760695.012567:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760695.012577:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.012590:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.012616:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760695.012647:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.013643:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624704:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760695.049547:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760695.049559:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.049568:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.049589:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760695.049615:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493624704:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.103374:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972366 gen 3 00000100:00100000:3.0:1789760695.247988:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972368 gen 3 00000008:00010000:3.0:1789760695.248039:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760695.248048:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.248061:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.248098:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760695.248133:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493625088:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.249567:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.249596:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493625216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.300892:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972369 gen 3 00000008:00010000:3.0:1789760695.300943:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760695.300952:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.300965:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.300991:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760695.301023:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493625216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.403875:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972371 gen 3 00000008:00010000:3.0:1789760695.403926:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760695.403934:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.403945:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.404003:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760695.404034:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493625472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.404963:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493625600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.453499:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972372 gen 3 00000008:00010000:3.0:1789760695.453551:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760695.453560:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.453573:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.453602:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760695.453632:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493625600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.549035:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972374 gen 3 00000008:00010000:3.0:1789760695.549093:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760695.549101:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.549112:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.549131:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760695.549169:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493625856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.550008:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.550027:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493625984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.603768:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972375 gen 3 00000008:00010000:3.0:1789760695.603837:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760695.603846:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.603858:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.603877:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760695.603907:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493625984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.605315:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493626112:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.678173:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972376 gen 3 00000008:00010000:3.0:1789760695.678225:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760695.678234:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.678246:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.678275:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760695.678307:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493626112:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760695.682504:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:3.0:1789760695.682623:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 20717568<>20721664 00000100:00100000:3.0:1789760695.682707:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493626240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.740217:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972377 gen 3 00000008:00010000:3.0:1789760695.740265:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760695.740273:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.740285:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.740304:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760695.740341:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493626240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.827224:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972378 gen 3 00000008:00010000:3.0:1789760695.827275:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760695.827284:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.827297:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.827320:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760695.827351:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493626368:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.828314:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.828331:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493626496:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.890877:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972379 gen 3 00000008:00010000:3.0:1789760695.890926:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760695.890934:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.890945:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.890967:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760695.891001:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493626496:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.891702:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627008:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.942582:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972380 gen 3 00000008:00010000:3.0:1789760695.942637:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760695.942647:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.942659:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.942734:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760695.942764:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627008:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.943630:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493627136:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.987090:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972381 gen 3 00000008:00010000:3.0:1789760695.987139:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760695.987148:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.987172:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.987189:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760695.987218:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493627136:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760695.988040:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627264:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.020039:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972382 gen 3 00000008:00010000:3.0:1789760696.020086:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760696.020094:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.020105:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.020122:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760696.020662:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627264:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760696.020713:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:3.0:1789760696.020874:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 20742144<>20746240 00000100:00100000:3.0:1789760696.055495:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972383 gen 3 00000008:00010000:3.0:1789760696.055545:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760696.055555:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.055567:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.055588:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760696.055614:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493627392:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.056456:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627520:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760696.108705:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760696.108721:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.108742:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.108760:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760696.108790:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627520:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.147144:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972385 gen 3 00000008:00010000:3.0:1789760696.147222:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760696.147231:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.147242:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.147263:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760696.147294:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627648:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.148193:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627776:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760696.188230:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760696.188245:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.188256:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.188274:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760696.188303:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627776:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.189165:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.189183:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.225333:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972387 gen 3 00000008:00010000:3.0:1789760696.225426:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760696.225435:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.225446:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.225465:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760696.225494:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493627904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.226330:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.271474:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972388 gen 3 00000008:00010000:3.0:1789760696.271529:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760696.271538:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.271551:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.271576:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760696.271605:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.272462:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493628160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.315976:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972389 gen 3 00000008:00010000:3.0:1789760696.316029:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760696.316037:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.316049:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.316070:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760696.316098:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493628160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.317270:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628288:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760696.364087:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760696.364104:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.364117:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.364128:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760696.364169:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.365189:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493628416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.403279:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972390 gen 3 00000100:00100000:3.0:1789760696.403332:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83480 x1876699493628416/t4294972391(4294972391) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760712 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760696.403388:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760696.403436:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.403448:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.403471:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760696.403496:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493628416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.451383:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.497872:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972393 gen 3 00000008:00010000:3.0:1789760696.497940:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760696.497949:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.497960:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.497983:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760696.498013:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493628672:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760696.556768:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760696.909859:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.909879:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629696:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760696.982238:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760696.982254:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.982268:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.982288:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760696.982319:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760696.983023:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.029290:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972402 gen 3 00000008:00010000:3.0:1789760697.029354:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760697.029363:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.029374:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.029422:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760697.029451:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.031421:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760697.031446:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.096097:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972403 gen 3 00000008:00010000:3.0:1789760697.096256:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760697.096266:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.096279:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.096301:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760697.096333:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493629952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.101232:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.148821:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972404 gen 3 00000008:00010000:3.0:1789760697.148870:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760697.148878:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.148890:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.148910:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760697.148949:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.150515:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760697.150538:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.228691:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972405 gen 3 00000008:00010000:3.0:1789760697.228743:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760697.228751:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.228772:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.228790:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760697.228817:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.229548:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.287379:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972406 gen 3 00000008:00010000:3.0:1789760697.287450:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760697.287458:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.287469:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.287489:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760697.287516:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.288455:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760697.288471:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630464:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760697.330651:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760697.330677:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.330690:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.330710:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760697.330744:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630464:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.332437:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.370123:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972408 gen 3 00000008:00010000:3.0:1789760697.370234:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760697.370246:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.370254:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.370278:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760697.370361:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493630592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.371220:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630720:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.430273:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972409 gen 3 00000008:00010000:3.0:1789760697.430314:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760697.430321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.430329:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.430347:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760697.430370:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630720:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.432650:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760697.432680:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.467974:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972410 gen 3 00000008:00010000:3.0:1789760697.468007:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760697.468013:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.468020:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.468038:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760697.468060:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.469587:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.527771:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972411 gen 3 00000008:00010000:3.0:1789760697.527804:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760697.527810:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.527822:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.527839:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760697.527861:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493630976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.528442:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631104:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.575011:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972412 gen 3 00000008:00010000:3.0:1789760697.575044:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760697.575050:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.575058:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.575081:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760697.575104:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631104:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.575648:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493631232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.612984:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972413 gen 3 00000008:00010000:3.0:1789760697.613015:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760697.613020:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.613028:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.613045:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760697.613068:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493631232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.613594:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631360:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760697.679327:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760697.679337:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.679347:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.679363:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760697.679428:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631360:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.680792:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760697.680809:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.719937:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972415 gen 3 00000008:00010000:3.0:1789760697.719991:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760697.719998:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.720007:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.720027:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760697.720050:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493631488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.721094:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760697.721109:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493631616:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760697.765648:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760697.765667:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.765678:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.765693:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760697.766337:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493631616:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760697.766367:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:3.0:1789760697.766526:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 20881408<>20885504 00000100:00100000:3.0:1789760697.766568:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493631744:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760697.805331:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972416 gen 3 00000100:00100000:3.0:1789760697.805370:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149c00 x1876699493631744/t4294972417(4294972417) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760713 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1789760698.001642:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972421 gen 3 00000008:00010000:3.0:1789760698.001689:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760698.001695:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.001705:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.001724:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760698.001748:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493632256:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760698.060420:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760698.060442:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493632512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760698.133355:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972423 gen 3 00000008:00010000:3.0:1789760698.133568:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760698.133577:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.133590:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.133611:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760698.133639:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493632512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760698.195677:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972424 gen 3 00000008:00010000:3.0:1789760698.195844:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760698.195852:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.195864:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.195890:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760698.195920:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493632640:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760698.272080:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5107, 5107] 00000080:00200000:3.0:1789760698.272096:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760698.272102:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760698.272118:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20922368 i_blocks 40967 00000020:00200000:3.0:1789760698.272122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760698.272133:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760698.272167:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20922368, count: 4096 00000080:00200000:3.0:1789760698.272175:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20922368, bytes: 4096 00000080:00200000:3.0:1789760698.272185:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760698.272194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760698.272197:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20922368, 20926464) 0 0 00000080:00200000:3.0:1789760698.272202:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760698.272219:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5108, 5108] 00020000:00200000:3.0:1789760698.272228:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20922368 chunk: [20922368, 20926464) 20926464, 4096 00020000:00200000:3.0:1789760698.272232:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760698.272257:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5108, 5108] 00000008:00010000:3.0:1789760698.272274:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760698.272281:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760698.272292:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20922368, 20926464) append 0 append lockpos 0 00000080:00200000:3.0:1789760698.272306:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20922368=0x13f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760698.272363:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760698.272366:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20926464 00000020:00010000:3.0:1789760698.272374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760698.272378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.272381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.272384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760698.272387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760698.272433:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760698.272441:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760698.272458:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760698.272570:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 20922368<>20926464 00000100:00100000:3.0:1789760698.272615:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760698.272630:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493633024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760698.318745:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972427 gen 3 00000008:00010000:3.0:1789760698.318798:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760698.318808:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.318820:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.318846:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760698.318878:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493633024:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760698.451429:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5109, 5109] 00000080:00200000:3.0:1789760698.451445:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760698.451452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760698.451468:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20930560 i_blocks 40967 00000020:00200000:3.0:1789760698.451472:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760698.451483:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760698.451518:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20930560, count: 4096 00000080:00200000:3.0:1789760698.451525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20930560, bytes: 4096 00000080:00200000:3.0:1789760698.451537:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760698.451545:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760698.451548:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20930560, 20934656) 0 0 00000080:00200000:3.0:1789760698.451553:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760698.451570:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5110, 5110] 00020000:00200000:3.0:1789760698.451579:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20930560 chunk: [20930560, 20934656) 20934656, 4096 00020000:00200000:3.0:1789760698.451584:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760698.451603:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5110, 5110] 00000008:00010000:3.0:1789760698.451623:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760698.451630:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760698.451643:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20930560, 20934656) append 0 append lockpos 0 00000080:00200000:3.0:1789760698.451668:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20930560=0x13f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760698.451742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760698.451745:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20934656 00000020:00010000:3.0:1789760698.451755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760698.451759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.451762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.451767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760698.451769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760698.451778:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760698.451784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760698.451801:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760698.451910:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 20930560<>20934656 00000100:00100000:3.0:1789760698.545491:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972429 gen 3 00000008:00010000:3.0:1789760698.545540:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760698.545548:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.545559:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.545584:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760698.545612:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493633280:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760698.612257:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5111, 5111] 00000080:00200000:3.0:1789760698.612275:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760698.612281:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760698.612295:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20938752 i_blocks 40967 00000020:00200000:3.0:1789760698.612299:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760698.612310:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760698.612344:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20938752, count: 4096 00000080:00200000:3.0:1789760698.612352:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20938752, bytes: 4096 00000080:00200000:3.0:1789760698.612362:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760698.612371:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760698.612373:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20938752, 20942848) 0 0 00000080:00200000:3.0:1789760698.612378:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760698.612436:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5112, 5112] 00020000:00200000:3.0:1789760698.612445:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20938752 chunk: [20938752, 20942848) 20942848, 4096 00020000:00200000:3.0:1789760698.612450:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760698.612471:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5112, 5112] 00000008:00010000:3.0:1789760698.612490:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760698.612496:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760698.612509:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20938752, 20942848) append 0 append lockpos 0 00000080:00200000:3.0:1789760698.612522:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20938752=0x13f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760698.612577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760698.612581:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20942848 00000020:00010000:3.0:1789760698.612589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760698.612592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.612595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760698.612599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760698.612603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760698.612610:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760698.612617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760698.612634:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760698.612744:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 20938752<>20942848 00000100:00100000:3.0:1789760698.612787:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760698.612804:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493633536:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760698.667131:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760698.667160:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.667182:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.667206:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760698.667237:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493633536:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760699.249089:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5123, 5123] 00000080:00200000:3.0:1789760699.249107:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.249114:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.249139:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20987904 i_blocks 43015 00000020:00200000:3.0:1789760699.249144:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.249157:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.249191:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20987904, count: 4096 00000080:00200000:3.0:1789760699.249198:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20987904, bytes: 4096 00000080:00200000:3.0:1789760699.249211:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.249219:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.249221:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20987904, 20992000) 0 0 00000080:00200000:3.0:1789760699.249225:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.249241:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5124, 5124] 00020000:00200000:3.0:1789760699.249250:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20987904 chunk: [20987904, 20992000) 20992000, 4096 00020000:00200000:3.0:1789760699.249255:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.249275:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5124, 5124] 00000008:00010000:3.0:1789760699.249293:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.249299:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760699.249311:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20987904, 20992000) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.249326:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20987904=0x1404000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.249386:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.249389:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20992000 00000020:00010000:3.0:1789760699.249435:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.249440:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.249443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.249446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.249449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.249458:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.249464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.249482:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760699.249587:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 20987904<>20992000 00000080:00200000:3.0:1789760699.311132:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5124, 5124] 00000080:00200000:3.0:1789760699.311151:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.311157:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.311173:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 20992000 i_blocks 43015 00000020:00200000:3.0:1789760699.311177:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.311186:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.311219:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 20992000, count: 4096 00000080:00200000:3.0:1789760699.311226:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 20992000, bytes: 4096 00000080:00200000:3.0:1789760699.311237:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.311246:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.311248:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [20992000, 20996096) 0 0 00000080:00200000:3.0:1789760699.311253:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.311269:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5125, 5125] 00020000:00200000:3.0:1789760699.311278:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 20992000 chunk: [20992000, 20996096) 20996096, 4096 00020000:00200000:3.0:1789760699.311283:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.311302:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5125, 5125] 00000008:00010000:3.0:1789760699.311319:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.311326:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760699.311337:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [20992000, 20996096) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.311352:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=20992000=0x1405000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.311685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.311688:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 20996096 00000020:00010000:3.0:1789760699.311698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.311702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.311705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.311708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.311711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.311720:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.311727:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.311744:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760699.312278:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 20992000<>20996096 00000080:00200000:3.0:1789760699.387688:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5126, 5126] 00000080:00200000:3.0:1789760699.387706:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.387712:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.387729:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21000192 i_blocks 43015 00000020:00200000:3.0:1789760699.387732:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.387742:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.387779:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21000192, count: 4096 00000080:00200000:3.0:1789760699.387790:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21000192, bytes: 4096 00000080:00200000:3.0:1789760699.387802:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.387810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.387813:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21000192, 21004288) 0 0 00000080:00200000:3.0:1789760699.387817:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.387835:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5127, 5127] 00020000:00200000:3.0:1789760699.387845:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21000192 chunk: [21000192, 21004288) 21004288, 4096 00020000:00200000:3.0:1789760699.387850:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.387871:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5127, 5127] 00000008:00010000:3.0:1789760699.387891:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.387898:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760699.387909:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21000192, 21004288) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.387925:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21000192=0x1407000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.387980:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.387983:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21004288 00000020:00010000:3.0:1789760699.387991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.387995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.387998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.388001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.388004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.388011:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.388018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.388033:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760699.388449:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 21000192<>21004288 00000100:00100000:3.0:1789760699.432881:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972445 gen 3 00000100:00100000:3.0:1789760699.432927:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514a680 x1876699493635456/t4294972446(4294972446) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760715 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760699.432990:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760699.432997:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760699.433007:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760699.433029:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760699.433054:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493635456:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760699.631588:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760700.087140:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493636480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.242601:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972454 gen 3 00000008:00010000:3.0:1789760700.242671:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760700.242681:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.242708:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.242726:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760700.242758:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493636480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.243711:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636608:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.386025:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972455 gen 3 00000008:00010000:3.0:1789760700.386076:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760700.386085:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.386098:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.386118:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760700.386157:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636608:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.386912:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760700.386930:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.545888:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972456 gen 3 00000008:00010000:3.0:1789760700.545941:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760700.545950:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.545961:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.545981:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760700.546011:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.551126:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.624032:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972457 gen 3 00000008:00010000:3.0:1789760700.624082:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760700.624092:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.624105:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.624240:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760700.624277:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493636864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.630516:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493636992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.672475:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972458 gen 3 00000008:00010000:3.0:1789760700.672526:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760700.672535:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.672546:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.672567:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760700.672596:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493636992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.673600:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760700.674265:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493637120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.733723:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972459 gen 3 00000008:00010000:3.0:1789760700.733886:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760700.733896:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760700.733909:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760700.733990:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760700.734026:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493637120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760700.934503:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5142, 5142] 00000080:00200000:3.0:1789760700.934522:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.934528:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.934546:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21065728 i_blocks 43015 00000020:00200000:3.0:1789760700.934551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.934562:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.934598:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21065728, count: 4096 00000080:00200000:3.0:1789760700.934606:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21065728, bytes: 4096 00000080:00200000:3.0:1789760700.934615:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.934626:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.934628:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21065728, 21069824) 0 0 00000080:00200000:3.0:1789760700.934643:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.934661:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5143, 5143] 00020000:00200000:3.0:1789760700.934679:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21065728 chunk: [21065728, 21069824) 21069824, 4096 00020000:00200000:3.0:1789760700.934685:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.934707:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5143, 5143] 00000008:00010000:3.0:1789760700.934725:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.934731:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760700.934742:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21065728, 21069824) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.934757:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21065728=0x1417000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.934836:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.934840:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21069824 00000020:00010000:3.0:1789760700.934850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.934854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.934858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.934861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.934864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.934872:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.934879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.934901:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760700.935022:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 21065728<>21069824 00000100:00100000:3.0:1789760700.935078:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493637504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760700.987683:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493637632:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760701.005244:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972462 gen 3 00000008:00010000:3.0:1789760701.005295:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760701.005304:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.005315:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.005428:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760701.005458:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493637504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760701.088672:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5144, 5144] 00000080:00200000:3.0:1789760701.088699:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.088705:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.088720:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21073920 i_blocks 43015 00000020:00200000:3.0:1789760701.088724:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.088735:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.088772:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21073920, count: 4096 00000080:00200000:3.0:1789760701.088779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21073920, bytes: 4096 00000080:00200000:3.0:1789760701.088790:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.088799:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.088801:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21073920, 21078016) 0 0 00000080:00200000:3.0:1789760701.088806:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.088835:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5145, 5145] 00020000:00200000:3.0:1789760701.088844:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21073920 chunk: [21073920, 21078016) 21078016, 4096 00020000:00200000:3.0:1789760701.088849:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.088868:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5145, 5145] 00000008:00010000:3.0:1789760701.088886:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.088893:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760701.088905:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21073920, 21078016) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.088920:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21073920=0x1419000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.088982:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.088985:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21078016 00000020:00010000:3.0:1789760701.088994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.088999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.089002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.089006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.089009:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.089017:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.089024:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.089040:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760701.089498:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 21073920<>21078016 00000100:00100000:3.0:1789760701.089543:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760701.089562:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.153021:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972464 gen 3 00000008:00010000:3.0:1789760701.153093:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760701.153102:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.153123:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.153148:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760701.153178:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.325375:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.388267:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972467 gen 3 00000008:00010000:3.0:1789760701.388335:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760701.388343:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.388354:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.388374:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760701.388541:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.389549:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760701.389565:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638656:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760701.457493:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760701.457509:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.457523:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.457547:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760701.457577:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493638656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.494844:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972469 gen 3 00000008:00010000:3.0:1789760701.494916:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760701.494925:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.494936:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.494958:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760701.494991:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493638784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.495757:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493638912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.539866:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972470 gen 3 00000008:00010000:3.0:1789760701.539920:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760701.539929:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.539942:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.539963:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760701.542547:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493638912:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760701.542615:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:3.0:1789760701.542763:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 21102592<>21106688 00000100:00100000:3.0:1789760701.542831:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493639040:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760701.602031:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760701.602049:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.602063:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.602095:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760701.602129:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493639040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.852973:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760701.852997:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493639808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760701.912444:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972477 gen 3 00000008:00010000:3.0:1789760701.912498:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760701.912508:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.912520:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.912549:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760701.912579:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493639808:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760701.975917:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5159, 5159] 00000080:00200000:3.0:1789760701.975933:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.975939:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.975954:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21135360 i_blocks 43015 00000020:00200000:3.0:1789760701.975958:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.975967:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.975998:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21135360, count: 4096 00000080:00200000:3.0:1789760701.976005:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21135360, bytes: 4096 00000080:00200000:3.0:1789760701.976014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.976023:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.976025:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21135360, 21139456) 0 0 00000080:00200000:3.0:1789760701.976030:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.976044:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5160, 5160] 00020000:00200000:3.0:1789760701.976051:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21135360 chunk: [21135360, 21139456) 21139456, 4096 00020000:00200000:3.0:1789760701.976056:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.976072:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5160, 5160] 00000008:00010000:3.0:1789760701.976089:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.976094:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760701.976114:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21135360, 21139456) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.976128:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21135360=0x1428000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.976179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.976182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21139456 00000020:00010000:3.0:1789760701.976189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.976193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.976196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.976200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.976202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.976209:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.976215:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.976229:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760701.976473:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21135360<>21139456 00000100:00100000:3.0:1789760702.032478:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972479 gen 3 00000008:00010000:3.0:1789760702.032531:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760702.032540:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.032551:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.032576:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760702.032608:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640064:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760702.083161:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5161, 5161] 00000080:00200000:3.0:1789760702.083178:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.083183:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.083199:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21143552 i_blocks 43015 00000020:00200000:3.0:1789760702.083203:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.083214:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.083248:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21143552, count: 4096 00000080:00200000:3.0:1789760702.083255:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21143552, bytes: 4096 00000080:00200000:3.0:1789760702.083277:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.083287:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.083289:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21143552, 21147648) 0 0 00000080:00200000:3.0:1789760702.083293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.083310:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5162, 5162] 00020000:00200000:3.0:1789760702.083318:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21143552 chunk: [21143552, 21147648) 21147648, 4096 00020000:00200000:3.0:1789760702.083323:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.083343:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5162, 5162] 00000008:00010000:3.0:1789760702.083360:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.083366:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760702.083378:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21143552, 21147648) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.083391:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21143552=0x142a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.083478:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.083481:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21147648 00000020:00010000:3.0:1789760702.083489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.083494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.083497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.083501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.083503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.083512:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.083521:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.083537:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760702.083652:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21143552<>21147648 00000100:00100000:3.0:1789760702.083693:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760702.083710:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760702.132191:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972481 gen 3 00000008:00010000:3.0:1789760702.132242:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760702.132251:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.132262:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.132282:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760702.132316:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640320:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760702.174559:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5163, 5163] 00000080:00200000:3.0:1789760702.174579:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.174585:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.174602:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21151744 i_blocks 43015 00000020:00200000:3.0:1789760702.174610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.174622:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.174675:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21151744, count: 4096 00000080:00200000:3.0:1789760702.174691:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21151744, bytes: 4096 00000080:00200000:3.0:1789760702.174701:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.174711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.174713:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21151744, 21155840) 0 0 00000080:00200000:3.0:1789760702.174718:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.174738:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5164, 5164] 00020000:00200000:3.0:1789760702.174746:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21151744 chunk: [21151744, 21155840) 21155840, 4096 00020000:00200000:3.0:1789760702.174749:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.174766:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5164, 5164] 00000008:00010000:3.0:1789760702.174780:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.174785:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760702.174795:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21151744, 21155840) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.174806:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21151744=0x142c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.174862:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.174864:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21155840 00000020:00010000:3.0:1789760702.174870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.174873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.174875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.174877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.174879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.174885:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.174890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.174902:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760702.174992:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21151744<>21155840 00000100:00100000:3.0:1789760702.237289:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972483 gen 3 00000008:00010000:3.0:1789760702.237335:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760702.237343:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.237353:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.237374:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760702.237441:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640576:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760702.311589:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5165, 5165] 00000080:00200000:3.0:1789760702.311607:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.311613:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.311638:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21159936 i_blocks 43015 00000020:00200000:3.0:1789760702.311642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.311654:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.311698:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21159936, count: 4096 00000080:00200000:3.0:1789760702.311706:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21159936, bytes: 4096 00000080:00200000:3.0:1789760702.311719:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.311730:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.311732:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21159936, 21164032) 0 0 00000080:00200000:3.0:1789760702.311737:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.311755:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5166, 5166] 00020000:00200000:3.0:1789760702.311763:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21159936 chunk: [21159936, 21164032) 21164032, 4096 00020000:00200000:3.0:1789760702.311768:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.311788:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5166, 5166] 00000008:00010000:3.0:1789760702.311806:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.311812:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760702.311834:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21159936, 21164032) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.311849:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21159936=0x142e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.311907:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.311910:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21164032 00000020:00010000:3.0:1789760702.311918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.311921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.311925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.311928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.311931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.311938:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.312169:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.312185:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760702.312286:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21159936<>21164032 00000100:00100000:3.0:1789760702.312332:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760702.312351:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640832:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760702.374453:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972485 gen 3 00000008:00010000:3.0:1789760702.374507:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760702.374518:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760702.374528:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760702.374557:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760702.374586:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493640832:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760702.700731:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760703.258191:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5178, 5178] 00000080:00200000:3.0:1789760703.258220:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.258228:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.258248:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21213184 i_blocks 43015 00000020:00200000:3.0:1789760703.258254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.258275:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.258309:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21213184, count: 4096 00000080:00200000:3.0:1789760703.258317:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21213184, bytes: 4096 00000080:00200000:3.0:1789760703.258332:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.258341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.258344:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21213184, 21217280) 0 0 00000080:00200000:3.0:1789760703.258348:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.258366:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5179, 5179] 00020000:00200000:3.0:1789760703.258375:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21213184 chunk: [21213184, 21217280) 21217280, 4096 00020000:00200000:3.0:1789760703.258380:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.258416:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5179, 5179] 00000008:00010000:3.0:1789760703.258436:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.258442:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760703.258455:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21213184, 21217280) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.258469:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21213184=0x143b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.259800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.259810:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21217280 00000020:00010000:3.0:1789760703.259824:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.259829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.259833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.259837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.259841:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.259854:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.259863:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.259909:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760703.260162:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21213184<>21217280 00000100:00100000:3.0:1789760703.260229:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760703.260258:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493642496:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760703.301594:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972498 gen 3 00000008:00010000:3.0:1789760703.301752:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760703.301762:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760703.301773:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760703.302324:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760703.302355:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493642496:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760703.338432:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5180, 5180] 00000080:00200000:3.0:1789760703.338451:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.338457:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.338473:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21221376 i_blocks 43015 00000020:00200000:3.0:1789760703.338477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.338487:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.338523:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21221376, count: 4096 00000080:00200000:3.0:1789760703.338531:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21221376, bytes: 4096 00000080:00200000:3.0:1789760703.338541:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.338549:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.338552:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21221376, 21225472) 0 0 00000080:00200000:3.0:1789760703.338557:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.338637:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5181, 5181] 00020000:00200000:3.0:1789760703.338647:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21221376 chunk: [21221376, 21225472) 21225472, 4096 00020000:00200000:3.0:1789760703.338651:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.338670:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5181, 5181] 00000008:00010000:3.0:1789760703.338732:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.338738:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760703.338751:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21221376, 21225472) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.338764:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21221376=0x143d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.338817:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.338819:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21225472 00000020:00010000:3.0:1789760703.338829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.338834:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.338837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.338840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.338844:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.338852:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.338859:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.338876:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760703.338977:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21221376<>21225472 00000100:00100000:3.0:1789760703.379709:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972500 gen 3 00000008:00010000:3.0:1789760703.379778:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760703.379787:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760703.379797:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760703.379872:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760703.379903:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493642752:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760703.705390:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5189, 5189] 00000080:00200000:3.0:1789760703.705432:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.705438:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.705456:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21258240 i_blocks 43015 00000020:00200000:3.0:1789760703.705460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.705472:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.705508:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21258240, count: 4096 00000080:00200000:3.0:1789760703.705515:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21258240, bytes: 4096 00000080:00200000:3.0:1789760703.705526:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.705537:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.705539:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21258240, 21262336) 0 0 00000080:00200000:3.0:1789760703.705544:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.705561:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5190, 5190] 00020000:00200000:3.0:1789760703.705570:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21258240 chunk: [21258240, 21262336) 21262336, 4096 00020000:00200000:3.0:1789760703.705575:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.705596:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5190, 5190] 00000008:00010000:3.0:1789760703.705699:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.705705:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760703.705717:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21258240, 21262336) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.705733:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21258240=0x1446000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.705795:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.705798:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21262336 00000020:00010000:3.0:1789760703.705807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.705811:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.705814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.705817:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.705820:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.705829:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.705836:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.705852:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760703.705968:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 21258240<>21262336 00000008:00010000:3.0:1789760703.746838:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760703.746854:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760703.746864:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760703.746935:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760703.746963:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493643904:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760703.880986:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5192, 5192] 00000080:00200000:3.0:1789760703.881005:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.881023:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.881040:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21270528 i_blocks 43015 00000020:00200000:3.0:1789760703.881044:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.881057:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.881112:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21270528, count: 4096 00000080:00200000:3.0:1789760703.881123:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21270528, bytes: 4096 00000080:00200000:3.0:1789760703.881138:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.881147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.881149:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21270528, 21274624) 0 0 00000080:00200000:3.0:1789760703.881153:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.881170:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5193, 5193] 00020000:00200000:3.0:1789760703.881190:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21270528 chunk: [21270528, 21274624) 21274624, 4096 00020000:00200000:3.0:1789760703.881195:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.881216:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5193, 5193] 00000008:00010000:3.0:1789760703.881235:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.881241:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760703.881252:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21270528, 21274624) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.881266:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21270528=0x1449000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.881323:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.881326:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21274624 00000020:00010000:3.0:1789760703.881335:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.881339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.881342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.881345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.881348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.881356:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.881363:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.881378:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760703.881533:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 21270528<>21274624 00000100:00100000:3.0:1789760703.961805:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972512 gen 3 00000008:00010000:3.0:1789760703.961860:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760703.961869:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760703.961881:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760703.961904:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760703.961934:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493644288:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760704.353624:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5199, 5199] 00000080:00200000:3.0:1789760704.353641:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760704.353647:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760704.353672:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21299200 i_blocks 43015 00000020:00200000:3.0:1789760704.353676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760704.353688:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760704.353723:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21299200, count: 4096 00000080:00200000:3.0:1789760704.353731:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21299200, bytes: 4096 00000080:00200000:3.0:1789760704.353743:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760704.353751:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760704.353753:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21299200, 21303296) 0 0 00000080:00200000:3.0:1789760704.353757:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760704.353773:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5200, 5200] 00020000:00200000:3.0:1789760704.353782:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21299200 chunk: [21299200, 21303296) 21303296, 4096 00020000:00200000:3.0:1789760704.353786:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760704.353806:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5200, 5200] 00000008:00010000:3.0:1789760704.353836:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760704.353842:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760704.353853:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21299200, 21303296) append 0 append lockpos 0 00000080:00200000:3.0:1789760704.353869:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21299200=0x1450000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760704.353928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760704.353931:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21303296 00000020:00010000:3.0:1789760704.353940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760704.353944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.353947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.353951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760704.353955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760704.353963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760704.353970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760704.353985:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760704.354086:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 21299200<>21303296 00000100:00100000:3.0:1789760704.354131:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760704.354147:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493645184:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760704.453236:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972519 gen 3 00000008:00010000:3.0:1789760704.453290:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760704.453298:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760704.453310:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760704.453418:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760704.453446:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493645184:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760704.702730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5205, 5205] 00000080:00200000:3.0:1789760704.702748:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760704.702754:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760704.702770:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21323776 i_blocks 43015 00000020:00200000:3.0:1789760704.702774:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760704.702785:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760704.702818:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21323776, count: 4096 00000080:00200000:3.0:1789760704.702825:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21323776, bytes: 4096 00000080:00200000:3.0:1789760704.702835:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760704.702843:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760704.702845:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21323776, 21327872) 0 0 00000080:00200000:3.0:1789760704.702849:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760704.702866:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5206, 5206] 00020000:00200000:3.0:1789760704.702875:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21323776 chunk: [21323776, 21327872) 21327872, 4096 00020000:00200000:3.0:1789760704.702879:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760704.702900:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5206, 5206] 00000008:00010000:3.0:1789760704.702918:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760704.702925:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760704.702936:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21323776, 21327872) append 0 append lockpos 0 00000080:00200000:3.0:1789760704.702952:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21323776=0x1456000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760704.703010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760704.703013:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21327872 00000020:00010000:3.0:1789760704.703022:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760704.703026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.703030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.703033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760704.703036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760704.703044:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760704.703051:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760704.703069:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760704.703190:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 21323776<>21327872 00000100:00100000:3.0:1789760704.808848:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972525 gen 3 00000008:00010000:3.0:1789760704.808900:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760704.808908:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760704.808918:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760704.808945:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760704.808974:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493645952:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760704.955071:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5208, 5208] 00000080:00200000:3.0:1789760704.955090:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760704.955096:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760704.955112:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21336064 i_blocks 43015 00000020:00200000:3.0:1789760704.955116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760704.955128:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760704.955162:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21336064, count: 4096 00000080:00200000:3.0:1789760704.955170:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21336064, bytes: 4096 00000080:00200000:3.0:1789760704.955181:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760704.955190:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760704.955192:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21336064, 21340160) 0 0 00000080:00200000:3.0:1789760704.955197:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760704.955214:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5209, 5209] 00020000:00200000:3.0:1789760704.955223:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21336064 chunk: [21336064, 21340160) 21340160, 4096 00020000:00200000:3.0:1789760704.955227:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760704.955248:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5209, 5209] 00000008:00010000:3.0:1789760704.955933:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760704.955941:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760704.955956:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21336064, 21340160) append 0 append lockpos 0 00000080:00200000:3.0:1789760704.955972:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21336064=0x1459000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760704.956036:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760704.956040:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21340160 00000020:00010000:3.0:1789760704.956049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760704.956052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.956056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.956059:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760704.956063:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760704.956072:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760704.956079:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760704.956099:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760704.956207:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 21336064<>21340160 00000100:00100000:3.0:1789760705.109746:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972528 gen 3 00000008:00010000:3.0:1789760705.109801:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760705.109811:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.109822:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.109905:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760705.109934:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493646336:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760705.603034:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5213, 5213] 00000080:00200000:3.0:1789760705.603050:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760705.603057:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760705.603073:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21356544 i_blocks 43015 00000020:00200000:3.0:1789760705.603077:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760705.603099:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760705.603133:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21356544, count: 4096 00000080:00200000:3.0:1789760705.603141:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21356544, bytes: 4096 00000080:00200000:3.0:1789760705.603154:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760705.603163:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760705.603166:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21356544, 21360640) 0 0 00000080:00200000:3.0:1789760705.603170:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760705.603186:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5214, 5214] 00020000:00200000:3.0:1789760705.603194:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21356544 chunk: [21356544, 21360640) 21360640, 4096 00020000:00200000:3.0:1789760705.603198:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760705.603218:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5214, 5214] 00000008:00010000:3.0:1789760705.603236:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760705.603242:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760705.603253:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21356544, 21360640) append 0 append lockpos 0 00000080:00200000:3.0:1789760705.603268:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21356544=0x145e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760705.603323:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760705.603325:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21360640 00000020:00010000:3.0:1789760705.603333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760705.603337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.603340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760705.603343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760705.603346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760705.603354:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760705.603360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760705.603377:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760705.603519:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 21356544<>21360640 00000100:00100000:3.0:1789760705.603567:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760705.603584:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493646976:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760705.775366:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760705.787619:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972532 gen 3 00000100:00100000:3.0:1789760705.787663:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149880 x1876699493646976/t4294972533(4294972533) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760721 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760705.787714:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:3.0:1789760705.787721:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760705.787732:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760705.787759:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:3.0:1789760705.787784:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493646976:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760706.973995:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5225, 5225] 00000080:00200000:3.0:1789760706.974014:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760706.974021:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760706.974037:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21405696 i_blocks 43015 00000020:00200000:3.0:1789760706.974040:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760706.974051:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760706.974095:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21405696, count: 4096 00000080:00200000:3.0:1789760706.974103:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21405696, bytes: 4096 00000080:00200000:3.0:1789760706.974117:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760706.974138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760706.974141:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21405696, 21409792) 0 0 00000080:00200000:3.0:1789760706.974155:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760706.974172:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5226, 5226] 00020000:00200000:3.0:1789760706.974180:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21405696 chunk: [21405696, 21409792) 21409792, 4096 00020000:00200000:3.0:1789760706.974184:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760706.974203:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5226, 5226] 00000008:00010000:3.0:1789760706.974231:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760706.974237:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760706.974249:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21405696, 21409792) append 0 append lockpos 0 00000080:00200000:3.0:1789760706.974262:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21405696=0x146a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760706.974321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760706.974324:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21409792 00000020:00010000:3.0:1789760706.974333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760706.974337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.974340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.974344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760706.974347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760706.974354:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760706.974362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760706.974379:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760706.974516:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 21405696<>21409792 00000080:00200000:3.0:1789760707.827050:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5233, 5233] 00000080:00200000:3.0:1789760707.827077:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.827084:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760707.827099:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21438464 i_blocks 43015 00000020:00200000:3.0:1789760707.827103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760707.827121:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760707.827163:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21438464, count: 4096 00000080:00200000:3.0:1789760707.827169:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21438464, bytes: 4096 00000080:00200000:3.0:1789760707.827179:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760707.827188:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760707.827190:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21438464, 21442560) 0 0 00000080:00200000:3.0:1789760707.827194:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760707.827212:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5234, 5234] 00020000:00200000:3.0:1789760707.827231:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21438464 chunk: [21438464, 21442560) 21442560, 4096 00020000:00200000:3.0:1789760707.827235:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760707.827254:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5234, 5234] 00000008:00010000:3.0:1789760707.827271:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760707.827277:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760707.827288:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21438464, 21442560) append 0 append lockpos 0 00000080:00200000:3.0:1789760707.827301:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21438464=0x1472000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760707.827355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760707.827358:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21442560 00000020:00010000:3.0:1789760707.827367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760707.827370:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.827374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.827377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760707.827380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760707.827387:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760707.827426:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760707.827442:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760707.827545:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21438464<>21442560 00000100:00100000:3.0:1789760707.827600:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.827616:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493649920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760707.873427:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972553 gen 3 00000008:00010000:3.0:1789760707.873487:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760707.873496:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.873505:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.873557:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760707.873594:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493649920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760707.879274:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493650048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760707.943206:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972554 gen 3 00000008:00010000:3.0:1789760707.943319:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760707.943327:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.943339:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.943361:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760707.944024:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493650048:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760707.944079:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:3.0:1789760707.944174:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21446656<>21450752 00000100:00100000:3.0:1789760707.944209:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.944336:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493650176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.039943:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972555 gen 3 00000008:00010000:3.0:1789760708.039995:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760708.040003:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.040015:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.040032:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760708.040066:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493650176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.163766:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972556 gen 3 00000008:00010000:3.0:1789760708.163822:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760708.163831:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.163844:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.163905:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760708.163933:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493650304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.166122:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493650432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.359442:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972559 gen 3 00000008:00010000:3.0:1789760708.359495:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760708.359503:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.359517:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.359546:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760708.359574:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493650688:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.437758:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493650944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.533686:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972561 gen 3 00000008:00010000:3.0:1789760708.533740:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760708.533749:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.533760:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.533781:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760708.533811:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493650944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.598520:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972562 gen 3 00000008:00010000:3.0:1789760708.598564:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760708.598571:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.598597:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.598616:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760708.598651:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651072:192.168.203.160@tcp:4:dd.0 00020000:00200000:2.0:1789760708.598919:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21479424 chunk: [21479424, 21483520) 21483520, 4096 00020000:00200000:2.0:1789760708.598925:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760708.598945:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5244, 5244] 00000008:00010000:2.0:1789760708.598964:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760708.598969:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760708.598981:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21479424, 21483520) append 0 append lockpos 0 00000080:00200000:2.0:1789760708.598995:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21479424=0x147c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760708.599059:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760708.599062:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21483520 00000020:00010000:2.0:1789760708.599070:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760708.599073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.599076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.599080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760708.599083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760708.599091:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760708.599097:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760708.599114:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760708.599362:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 21479424<>21483520 00000100:00100000:3.0:1789760708.599517:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.672327:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972563 gen 3 00000008:00010000:3.0:1789760708.672384:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760708.672408:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.672423:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.672452:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760708.672484:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760708.672758:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5244, 5244] 00000080:00200000:2.0:1789760708.672775:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760708.672782:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760708.672798:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21483520 i_blocks 43015 00000020:00200000:2.0:1789760708.672801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760708.672813:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760708.672850:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21483520, count: 4096 00000080:00200000:2.0:1789760708.672857:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21483520, bytes: 4096 00000080:00200000:2.0:1789760708.672872:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760708.672881:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760708.672897:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21483520, 21487616) 0 0 00000080:00200000:2.0:1789760708.672902:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760708.672920:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5245, 5245] 00020000:00200000:2.0:1789760708.672929:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21483520 chunk: [21483520, 21487616) 21487616, 4096 00020000:00200000:2.0:1789760708.672935:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760708.672957:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5245, 5245] 00000008:00010000:2.0:1789760708.672976:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760708.672982:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760708.672994:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21483520, 21487616) append 0 append lockpos 0 00000080:00200000:2.0:1789760708.673009:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21483520=0x147d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760708.673067:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760708.673070:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21487616 00000020:00010000:2.0:1789760708.673078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760708.673081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.673084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.673088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760708.673091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760708.673113:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760708.673121:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760708.673138:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760708.673259:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 21483520<>21487616 00000100:00100000:3.0:1789760708.673531:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760708.673552:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651328:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.770783:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972564 gen 3 00000008:00010000:3.0:1789760708.770834:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760708.770843:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.770856:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.770876:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760708.770905:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760708.771007:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5245, 5245] 00000080:00200000:2.0:1789760708.771025:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760708.771031:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760708.771046:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21487616 i_blocks 43015 00000020:00200000:2.0:1789760708.771050:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760708.771071:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760708.771332:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21487616, count: 4096 00000080:00200000:2.0:1789760708.771341:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21487616, bytes: 4096 00000080:00200000:2.0:1789760708.771350:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760708.771359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760708.771361:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21487616, 21491712) 0 0 00000080:00200000:2.0:1789760708.771366:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760708.771383:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5246, 5246] 00020000:00200000:2.0:1789760708.771425:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21487616 chunk: [21487616, 21491712) 21491712, 4096 00020000:00200000:2.0:1789760708.771433:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760708.771454:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5246, 5246] 00000008:00010000:2.0:1789760708.771471:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760708.771477:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760708.771489:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21487616, 21491712) append 0 append lockpos 0 00000080:00200000:2.0:1789760708.771503:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21487616=0x147e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760708.771557:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760708.771560:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21491712 00000020:00010000:2.0:1789760708.771568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760708.771571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.771586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.771590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760708.771593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760708.771600:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760708.771607:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760708.771625:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760708.771731:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 21487616<>21491712 00000100:00100000:3.0:1789760708.771844:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651456:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760708.848092:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760708.863172:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972565 gen 3 00000008:00010000:3.0:1789760708.863237:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760708.863246:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.863258:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.863277:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760708.863306:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760708.863750:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5246, 5246] 00000080:00200000:2.0:1789760708.863767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760708.863773:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760708.863789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21491712 i_blocks 43015 00000020:00200000:2.0:1789760708.863794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760708.863804:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760708.863837:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21491712, count: 4096 00000080:00200000:2.0:1789760708.863845:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21491712, bytes: 4096 00000080:00200000:2.0:1789760708.863854:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760708.863863:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760708.863866:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21491712, 21495808) 0 0 00000080:00200000:2.0:1789760708.863871:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760708.863887:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5247, 5247] 00020000:00200000:2.0:1789760708.863896:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21491712 chunk: [21491712, 21495808) 21495808, 4096 00020000:00200000:2.0:1789760708.863902:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760708.863923:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5247, 5247] 00000008:00010000:2.0:1789760708.863942:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760708.863948:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760708.863961:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21491712, 21495808) append 0 append lockpos 0 00000080:00200000:2.0:1789760708.863975:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21491712=0x147f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760708.864034:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760708.864037:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21495808 00000020:00010000:2.0:1789760708.864046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760708.864049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.864052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.864056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760708.864058:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760708.864066:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760708.864072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760708.864087:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760708.864359:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 21491712<>21495808 00000100:00100000:3.0:1789760708.864726:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760708.864750:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651584:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760708.934779:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972566 gen 3 00000008:00010000:3.0:1789760708.934844:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760708.934853:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.934864:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.934881:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760708.934920:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760708.935634:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5247, 5247] 00000080:00200000:2.0:1789760708.935651:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760708.935657:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760708.935673:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21495808 i_blocks 43015 00000020:00200000:2.0:1789760708.935678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760708.935689:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760708.935721:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21495808, count: 4096 00000080:00200000:2.0:1789760708.935728:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21495808, bytes: 4096 00000080:00200000:2.0:1789760708.935740:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760708.935749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760708.935752:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21495808, 21499904) 0 0 00000080:00200000:2.0:1789760708.935756:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760708.935775:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5248, 5248] 00020000:00200000:2.0:1789760708.935783:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21495808 chunk: [21495808, 21499904) 21499904, 4096 00020000:00200000:2.0:1789760708.935789:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760708.935810:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5248, 5248] 00000008:00010000:2.0:1789760708.935841:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760708.935848:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760708.935860:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21495808, 21499904) append 0 append lockpos 0 00000080:00200000:2.0:1789760708.935874:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21495808=0x1480000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760708.935933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760708.935936:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21499904 00000020:00010000:2.0:1789760708.935943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760708.935946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.935949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760708.935953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760708.935955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760708.935963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760708.935969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760708.935986:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760708.936469:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 21495808<>21499904 00000100:00100000:3.0:1789760708.936697:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.002842:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972567 gen 3 00000008:00010000:3.0:1789760709.002900:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760709.002909:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.002922:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.002946:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760709.002974:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760709.003176:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5248, 5248] 00000080:00200000:2.0:1789760709.003193:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760709.003200:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760709.003364:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21499904 i_blocks 43015 00000020:00200000:2.0:1789760709.003369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760709.003380:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760709.003570:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21499904, count: 4096 00000080:00200000:2.0:1789760709.003588:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21499904, bytes: 4096 00000080:00200000:2.0:1789760709.003599:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760709.003607:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760709.003609:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21499904, 21504000) 0 0 00000080:00200000:2.0:1789760709.003614:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760709.003639:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5249, 5249] 00020000:00200000:2.0:1789760709.003648:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21499904 chunk: [21499904, 21504000) 21504000, 4096 00020000:00200000:2.0:1789760709.003654:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760709.003674:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5249, 5249] 00000008:00010000:2.0:1789760709.003691:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760709.003696:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760709.003707:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21499904, 21504000) append 0 append lockpos 0 00000080:00200000:2.0:1789760709.003724:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21499904=0x1481000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760709.003782:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760709.003785:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21504000 00000020:00010000:2.0:1789760709.003793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760709.003796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.003799:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.003803:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760709.003806:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760709.003814:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760709.003820:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760709.003838:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760709.007106:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 21499904<>21504000 00000100:00100000:3.0:1789760709.007750:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760709.007783:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.052895:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972568 gen 3 00000008:00010000:3.0:1789760709.052942:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760709.052951:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.052963:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.053032:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760709.053072:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760709.054612:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5249, 5249] 00000080:00200000:2.0:1789760709.054640:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760709.054647:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760709.054663:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21504000 i_blocks 43015 00000020:00200000:2.0:1789760709.054667:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760709.054681:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760709.054713:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21504000, count: 4096 00000080:00200000:2.0:1789760709.054720:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21504000, bytes: 4096 00000080:00200000:2.0:1789760709.054730:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760709.054738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760709.054740:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21504000, 21508096) 0 0 00000080:00200000:2.0:1789760709.054744:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760709.054759:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5250, 5250] 00020000:00200000:2.0:1789760709.054767:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21504000 chunk: [21504000, 21508096) 21508096, 4096 00020000:00200000:2.0:1789760709.054772:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760709.054789:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5250, 5250] 00000008:00010000:2.0:1789760709.054805:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760709.054810:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760709.054831:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21504000, 21508096) append 0 append lockpos 0 00000080:00200000:2.0:1789760709.054845:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21504000=0x1482000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760709.054902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760709.054905:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21508096 00000020:00010000:2.0:1789760709.054913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760709.054916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.054919:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.054922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760709.054925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760709.054932:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760709.054938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760709.054961:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760709.055073:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 21504000<>21508096 00000100:00100000:3.0:1789760709.055158:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651968:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.104362:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972569 gen 3 00000008:00010000:3.0:1789760709.104459:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760709.104468:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.104480:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.104500:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760709.104523:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493651968:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760709.104649:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5250, 5250] 00000080:00200000:2.0:1789760709.104665:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760709.104672:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760709.104688:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21508096 i_blocks 43015 00000020:00200000:2.0:1789760709.104692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760709.104702:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760709.104736:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21508096, count: 4096 00000080:00200000:2.0:1789760709.104743:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21508096, bytes: 4096 00000080:00200000:2.0:1789760709.104753:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760709.104761:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760709.104763:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21508096, 21512192) 0 0 00000080:00200000:2.0:1789760709.104768:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760709.104785:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5251, 5251] 00020000:00200000:2.0:1789760709.104794:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21508096 chunk: [21508096, 21512192) 21512192, 4096 00020000:00200000:2.0:1789760709.104800:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760709.104819:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5251, 5251] 00000008:00010000:2.0:1789760709.104837:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760709.104843:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760709.104854:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21508096, 21512192) append 0 append lockpos 0 00000080:00200000:2.0:1789760709.104868:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21508096=0x1483000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760709.104922:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760709.104925:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21512192 00000020:00010000:2.0:1789760709.105690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760709.105699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.105703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.105707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760709.105710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760709.105721:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760709.105729:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760709.105750:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760709.105872:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 21508096<>21512192 00000100:00100000:3.0:1789760709.105954:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493652096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.144297:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972570 gen 3 00000008:00010000:3.0:1789760709.144344:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760709.144352:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.144363:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.144383:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760709.144446:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493652096:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760709.144738:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5251, 5251] 00000080:00200000:2.0:1789760709.144753:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760709.144759:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760709.144776:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21512192 i_blocks 43015 00000020:00200000:2.0:1789760709.144779:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760709.144790:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760709.144888:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21512192, count: 4096 00000080:00200000:2.0:1789760709.144896:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21512192, bytes: 4096 00000080:00200000:2.0:1789760709.144906:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760709.144915:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760709.144917:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21512192, 21516288) 0 0 00000080:00200000:2.0:1789760709.144921:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760709.144937:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5252, 5252] 00020000:00200000:2.0:1789760709.144945:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21512192 chunk: [21512192, 21516288) 21516288, 4096 00020000:00200000:2.0:1789760709.144950:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760709.144969:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5252, 5252] 00000008:00010000:2.0:1789760709.144987:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760709.144993:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760709.145004:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21512192, 21516288) append 0 append lockpos 0 00000080:00200000:2.0:1789760709.145018:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21512192=0x1484000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760709.145073:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760709.145076:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21516288 00000020:00010000:2.0:1789760709.145084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760709.145087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.145090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760709.145094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760709.145097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760709.145104:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760709.145110:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760709.145128:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760709.145224:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21512192<>21516288 00000100:00100000:3.0:1789760709.145296:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493652224:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.200009:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972571 gen 3 00000008:00010000:3.0:1789760709.200069:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760709.200078:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.200091:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.200113:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760709.200145:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493652224:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760709.788694:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5262, 5262] 00000080:00200000:3.0:1789760709.788715:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760709.788720:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760709.788733:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21557248 i_blocks 43015 00000020:00200000:3.0:1789760709.788736:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760709.788744:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760709.788769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21557248, count: 4096 00000080:00200000:3.0:1789760709.788775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21557248, bytes: 4096 00000080:00200000:3.0:1789760709.788782:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760709.788790:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760709.788792:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21557248, 21561344) 0 0 00000080:00200000:3.0:1789760709.788796:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760709.788808:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5263, 5263] 00020000:00200000:3.0:1789760709.788814:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21557248 chunk: [21557248, 21561344) 21561344, 4096 00020000:00200000:3.0:1789760709.788818:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760709.788831:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5263, 5263] 00000008:00010000:3.0:1789760709.788843:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760709.788848:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760709.788856:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21557248, 21561344) append 0 append lockpos 0 00000080:00200000:3.0:1789760709.788866:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21557248=0x148f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760709.788905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760709.788907:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21561344 00000020:00010000:3.0:1789760709.788913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760709.788916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.788924:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.788928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760709.788931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760709.788937:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760709.788942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760709.788964:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760709.789037:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21557248<>21561344 00000100:00100000:3.0:1789760709.789075:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760709.789087:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493653632:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760709.883699:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972582 gen 3 00000008:00010000:3.0:1789760709.883745:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760709.883753:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.883764:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.883800:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760709.883838:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493653632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760709.945215:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5264, 5264] 00000080:00200000:3.0:1789760709.945232:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760709.945238:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760709.945253:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21565440 i_blocks 43015 00000020:00200000:3.0:1789760709.945257:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760709.945268:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760709.945303:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21565440, count: 4096 00000080:00200000:3.0:1789760709.945311:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21565440, bytes: 4096 00000080:00200000:3.0:1789760709.945321:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760709.945330:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760709.945333:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21565440, 21569536) 0 0 00000080:00200000:3.0:1789760709.945337:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760709.945353:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5265, 5265] 00020000:00200000:3.0:1789760709.945361:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21565440 chunk: [21565440, 21569536) 21569536, 4096 00020000:00200000:3.0:1789760709.945365:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760709.945383:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5265, 5265] 00000008:00010000:3.0:1789760709.945427:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760709.945433:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760709.945442:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21565440, 21569536) append 0 append lockpos 0 00000080:00200000:3.0:1789760709.945455:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21565440=0x1491000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760709.945522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760709.945525:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21569536 00000020:00010000:3.0:1789760709.945533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760709.945537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.945540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.945543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760709.945546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760709.945553:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760709.945559:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760709.945583:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760709.945681:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21565440<>21569536 00000100:00100000:2.0:1789760709.945744:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493653888:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.007037:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972584 gen 3 00000008:00010000:3.0:1789760710.007102:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760710.007110:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.007121:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.007143:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760710.007171:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493653888:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.075668:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5266, 5266] 00000080:00200000:2.0:1789760710.075685:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.075690:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.075706:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21573632 i_blocks 43015 00000020:00200000:2.0:1789760710.075709:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.075719:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.075750:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21573632, count: 4096 00000080:00200000:2.0:1789760710.075756:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21573632, bytes: 4096 00000080:00200000:2.0:1789760710.075770:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.075778:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.075781:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21573632, 21577728) 0 0 00000080:00200000:2.0:1789760710.075785:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.075800:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5267, 5267] 00020000:00200000:2.0:1789760710.075808:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21573632 chunk: [21573632, 21577728) 21577728, 4096 00020000:00200000:2.0:1789760710.075813:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.075844:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5267, 5267] 00000008:00010000:2.0:1789760710.075861:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.075866:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.075877:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21573632, 21577728) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.075890:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21573632=0x1493000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.075948:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.075951:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21577728 00000020:00010000:2.0:1789760710.075959:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.075962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.075965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.075969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.075972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.075979:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.075985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.075998:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760710.076215:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 21573632<>21577728 00000100:00100000:2.0:1789760710.076278:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493654144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.141192:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972586 gen 3 00000008:00010000:3.0:1789760710.141238:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760710.141246:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.141258:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.141277:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760710.141303:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493654144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.141433:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5267, 5267] 00000080:00200000:2.0:1789760710.141450:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.141456:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.141471:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21577728 i_blocks 43015 00000020:00200000:2.0:1789760710.141475:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.141485:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.141528:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21577728, count: 4096 00000080:00200000:2.0:1789760710.141535:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21577728, bytes: 4096 00000080:00200000:2.0:1789760710.141543:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.141552:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.141555:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21577728, 21581824) 0 0 00000080:00200000:2.0:1789760710.141559:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.141582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5268, 5268] 00020000:00200000:2.0:1789760710.141591:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21577728 chunk: [21577728, 21581824) 21581824, 4096 00020000:00200000:2.0:1789760710.141595:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.141614:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5268, 5268] 00000008:00010000:2.0:1789760710.141631:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.141638:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.141648:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21577728, 21581824) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.141661:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21577728=0x1494000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.141715:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.141723:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21581824 00000020:00010000:2.0:1789760710.141731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.141735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.141738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.141742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.141745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.141753:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.141759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.141773:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760710.141869:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 21577728<>21581824 00000100:00100000:3.0:1789760710.141935:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493654272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.204615:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972587 gen 3 00000008:00010000:3.0:1789760710.204673:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760710.204682:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.204694:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.204757:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760710.204785:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493654272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.205071:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5268, 5268] 00000080:00200000:2.0:1789760710.205085:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.205092:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.205107:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21581824 i_blocks 43015 00000020:00200000:2.0:1789760710.205111:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.205121:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.205152:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21581824, count: 4096 00000080:00200000:2.0:1789760710.205158:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21581824, bytes: 4096 00000080:00200000:2.0:1789760710.205168:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.205176:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.205179:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21581824, 21585920) 0 0 00000080:00200000:2.0:1789760710.205183:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.205200:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5269, 5269] 00020000:00200000:2.0:1789760710.205209:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21581824 chunk: [21581824, 21585920) 21585920, 4096 00020000:00200000:2.0:1789760710.205213:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.205231:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5269, 5269] 00000008:00010000:2.0:1789760710.205248:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.205254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.205265:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21581824, 21585920) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.205279:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21581824=0x1495000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.205331:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.205334:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21585920 00000020:00010000:2.0:1789760710.205342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.205346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.205349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.205352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.205355:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.205362:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.205368:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.205386:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760710.205609:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 21581824<>21585920 00000100:00100000:3.0:1789760710.205669:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760710.205876:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493654400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.271006:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972588 gen 3 00000008:00010000:3.0:1789760710.271059:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760710.271067:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.271080:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.271107:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760710.271146:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493654400:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760710.444501:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5272, 5272] 00000080:00200000:3.0:1789760710.444516:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760710.444523:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760710.444537:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21598208 i_blocks 43015 00000020:00200000:3.0:1789760710.444541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760710.444550:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760710.444598:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21598208, count: 4096 00000080:00200000:3.0:1789760710.444605:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21598208, bytes: 4096 00000080:00200000:3.0:1789760710.444623:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760710.444632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760710.444634:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21598208, 21602304) 0 0 00000080:00200000:3.0:1789760710.444639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760710.444654:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5273, 5273] 00020000:00200000:3.0:1789760710.444661:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21598208 chunk: [21598208, 21602304) 21602304, 4096 00020000:00200000:3.0:1789760710.444666:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760710.444684:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5273, 5273] 00000008:00010000:3.0:1789760710.444711:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760710.444717:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760710.444728:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21598208, 21602304) append 0 append lockpos 0 00000080:00200000:3.0:1789760710.444744:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21598208=0x1499000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760710.444797:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760710.444800:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21602304 00000020:00010000:3.0:1789760710.444808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760710.444812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760710.444826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760710.444830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760710.444832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760710.444840:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760710.444847:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760710.444863:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760710.444972:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21598208<>21602304 00000100:00100000:3.0:1789760710.445012:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760710.445030:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493654912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.534362:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972592 gen 3 00000008:00010000:3.0:1789760710.534452:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760710.534462:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.534472:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.534501:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760710.534530:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493654912:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760710.682242:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5276, 5276] 00000080:00200000:3.0:1789760710.682260:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760710.682265:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760710.682282:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21614592 i_blocks 43015 00000020:00200000:3.0:1789760710.682286:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760710.682296:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760710.682328:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21614592, count: 4096 00000080:00200000:3.0:1789760710.682335:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21614592, bytes: 4096 00000080:00200000:3.0:1789760710.682347:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760710.682356:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760710.682358:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21614592, 21618688) 0 0 00000080:00200000:3.0:1789760710.682363:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760710.682380:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5277, 5277] 00020000:00200000:3.0:1789760710.682390:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21614592 chunk: [21614592, 21618688) 21618688, 4096 00020000:00200000:3.0:1789760710.682419:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760710.682439:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5277, 5277] 00000008:00010000:3.0:1789760710.682458:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760710.682463:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760710.682474:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21614592, 21618688) append 0 append lockpos 0 00000080:00200000:3.0:1789760710.682487:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21614592=0x149d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760710.682537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760710.682539:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21618688 00000020:00010000:3.0:1789760710.682547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760710.682551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760710.682554:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760710.682572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760710.682575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760710.682583:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760710.682590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760710.682605:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760710.682708:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21614592<>21618688 00000100:00100000:3.0:1789760710.682753:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493655424:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.718774:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972596 gen 3 00000008:00010000:3.0:1789760710.718834:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760710.718842:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.718851:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.718870:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760710.718898:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493655424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.775792:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5278, 5278] 00000080:00200000:2.0:1789760710.775809:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.775815:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.775839:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21622784 i_blocks 43015 00000020:00200000:2.0:1789760710.775843:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.775853:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.775886:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21622784, count: 4096 00000080:00200000:2.0:1789760710.775894:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21622784, bytes: 4096 00000080:00200000:2.0:1789760710.775905:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.775914:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.775917:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21622784, 21626880) 0 0 00000080:00200000:2.0:1789760710.775921:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.775936:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5279, 5279] 00020000:00200000:2.0:1789760710.775944:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21622784 chunk: [21622784, 21626880) 21626880, 4096 00020000:00200000:2.0:1789760710.775949:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.775985:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5279, 5279] 00000008:00010000:2.0:1789760710.776003:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.776009:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.776020:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21622784, 21626880) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.776033:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21622784=0x149f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.776252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.776256:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21626880 00000020:00010000:2.0:1789760710.776264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.776268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.776271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.776275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.776278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.776286:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.776292:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.776307:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760710.776454:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 21622784<>21626880 00000100:00100000:3.0:1789760710.776833:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760710.776855:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493655680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.814549:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972598 gen 3 00000008:00010000:3.0:1789760710.814611:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760710.814630:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.814642:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.814662:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760710.814693:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493655680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.814833:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5279, 5279] 00000080:00200000:2.0:1789760710.814849:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.814855:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.814870:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21626880 i_blocks 43015 00000020:00200000:2.0:1789760710.814874:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.814885:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.814918:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21626880, count: 4096 00000080:00200000:2.0:1789760710.814926:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21626880, bytes: 4096 00000080:00200000:2.0:1789760710.814936:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.814944:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.814947:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21626880, 21630976) 0 0 00000080:00200000:2.0:1789760710.814965:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.814983:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5280, 5280] 00020000:00200000:2.0:1789760710.814992:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21626880 chunk: [21626880, 21630976) 21630976, 4096 00020000:00200000:2.0:1789760710.814997:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.815018:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5280, 5280] 00000008:00010000:2.0:1789760710.815036:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.815054:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.815068:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21626880, 21630976) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.815082:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21626880=0x14a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.815145:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.815148:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21630976 00000020:00010000:2.0:1789760710.815157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.815160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.815163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.815167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.815170:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.815178:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.815185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.815202:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760710.815305:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 21626880<>21630976 00000100:00100000:3.0:1789760710.815568:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760710.815588:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493655808:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760710.860299:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760710.860314:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.860327:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.860348:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760710.860377:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493655808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.860496:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5280, 5280] 00000080:00200000:2.0:1789760710.860512:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.860518:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.860532:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21630976 i_blocks 43015 00000020:00200000:2.0:1789760710.860536:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.860546:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.860595:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21630976, count: 4096 00000080:00200000:2.0:1789760710.860603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21630976, bytes: 4096 00000080:00200000:2.0:1789760710.860613:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.860622:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.860626:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21630976, 21635072) 0 0 00000080:00200000:2.0:1789760710.860630:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.860646:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5281, 5281] 00020000:00200000:2.0:1789760710.860654:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21630976 chunk: [21630976, 21635072) 21635072, 4096 00020000:00200000:2.0:1789760710.860659:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.860689:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5281, 5281] 00000008:00010000:2.0:1789760710.860706:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.860711:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.860721:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21630976, 21635072) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.860734:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21630976=0x14a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.860788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.860791:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21635072 00000020:00010000:2.0:1789760710.861026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.861030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.861034:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.861037:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.861049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.861057:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.861064:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.861081:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760710.861184:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 21630976<>21635072 00000100:00100000:3.0:1789760710.861256:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493655936:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.909836:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972599 gen 3 00000100:00100000:3.0:1789760710.910010:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81c00 x1876699493655936/t4294972600(4294972600) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760726 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760710.910778:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760710.910787:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.910798:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.910835:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760710.910867:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493655936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.911176:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5281, 5281] 00000080:00200000:2.0:1789760710.911192:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.911198:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.911213:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21635072 i_blocks 43015 00000020:00200000:2.0:1789760710.911217:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.911228:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.911262:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21635072, count: 4096 00000080:00200000:2.0:1789760710.911269:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21635072, bytes: 4096 00000080:00200000:2.0:1789760710.911284:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.911292:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.911295:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21635072, 21639168) 0 0 00000080:00200000:2.0:1789760710.911300:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.911315:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5282, 5282] 00020000:00200000:2.0:1789760710.911323:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21635072 chunk: [21635072, 21639168) 21639168, 4096 00020000:00200000:2.0:1789760710.911327:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.911347:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5282, 5282] 00000008:00010000:2.0:1789760710.911367:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.911373:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.911386:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21635072, 21639168) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.911424:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21635072=0x14a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.911482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.911485:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21639168 00000020:00010000:2.0:1789760710.911493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.911497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.911500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.911503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.911507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.911515:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.911522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.911538:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760710.911650:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 21635072<>21639168 00000100:00100000:3.0:1789760710.914092:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493656064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760710.956548:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972601 gen 3 00000008:00010000:3.0:1789760710.956640:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760710.956649:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.956662:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.956728:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760710.956760:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493656064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760710.957688:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5282, 5282] 00000080:00200000:2.0:1789760710.957708:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760710.957715:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760710.957732:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21639168 i_blocks 43015 00000020:00200000:2.0:1789760710.957737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760710.957750:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760710.957786:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21639168, count: 4096 00000080:00200000:2.0:1789760710.957796:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21639168, bytes: 4096 00000080:00200000:2.0:1789760710.957806:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760710.957814:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760710.957861:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21639168, 21643264) 0 0 00000080:00200000:2.0:1789760710.957866:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760710.958016:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5283, 5283] 00020000:00200000:2.0:1789760710.958026:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21639168 chunk: [21639168, 21643264) 21643264, 4096 00020000:00200000:2.0:1789760710.958030:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760710.958111:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5283, 5283] 00000008:00010000:2.0:1789760710.958132:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760710.958139:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760710.958153:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21639168, 21643264) append 0 append lockpos 0 00000080:00200000:2.0:1789760710.958169:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21639168=0x14a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760710.958235:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760710.958239:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21643264 00000020:00010000:2.0:1789760710.958247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760710.958250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.958254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760710.958257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760710.958260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760710.958268:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760710.958276:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760710.958299:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760710.958442:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 21639168<>21643264 00000100:00100000:3.0:1789760710.958528:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.001510:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972602 gen 3 00000008:00010000:3.0:1789760711.001575:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760711.001583:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.001593:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.001620:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760711.001649:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.001809:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5283, 5283] 00000080:00200000:2.0:1789760711.001836:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.001842:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.001859:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21643264 i_blocks 43015 00000020:00200000:2.0:1789760711.001863:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.001875:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.001911:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21643264, count: 4096 00000080:00200000:2.0:1789760711.001919:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21643264, bytes: 4096 00000080:00200000:2.0:1789760711.001930:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.001940:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.001943:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21643264, 21647360) 0 0 00000080:00200000:2.0:1789760711.001948:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.001965:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5284, 5284] 00020000:00200000:2.0:1789760711.001975:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21643264 chunk: [21643264, 21647360) 21647360, 4096 00020000:00200000:2.0:1789760711.001980:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.002001:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5284, 5284] 00000008:00010000:2.0:1789760711.002020:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.002026:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.002038:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21643264, 21647360) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.002177:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21643264=0x14a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.002237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.002240:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21647360 00000020:00010000:2.0:1789760711.002248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.002252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.002255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.002258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.002261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.002269:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.002275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.002291:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760711.003034:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 21643264<>21647360 00000100:00100000:3.0:1789760711.003158:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760711.003241:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.044629:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972603 gen 3 00000008:00010000:3.0:1789760711.044683:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760711.044692:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.044704:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.044718:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760711.044748:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656320:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760711.191608:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5287, 5287] 00000080:00200000:3.0:1789760711.191687:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760711.191693:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760711.191709:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21659648 i_blocks 43015 00000020:00200000:3.0:1789760711.191712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760711.191723:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760711.191756:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21659648, count: 4096 00000080:00200000:3.0:1789760711.191763:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21659648, bytes: 4096 00000080:00200000:3.0:1789760711.191774:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760711.191783:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760711.191785:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21659648, 21663744) 0 0 00000080:00200000:3.0:1789760711.191790:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760711.191806:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5288, 5288] 00020000:00200000:3.0:1789760711.192327:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21659648 chunk: [21659648, 21663744) 21663744, 4096 00020000:00200000:3.0:1789760711.192336:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760711.192360:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5288, 5288] 00000008:00010000:3.0:1789760711.192379:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760711.192386:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760711.192425:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21659648, 21663744) append 0 append lockpos 0 00000080:00200000:3.0:1789760711.192441:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21659648=0x14a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760711.192503:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760711.192506:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21663744 00000020:00010000:3.0:1789760711.192515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760711.192518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.192520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.192524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760711.192527:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760711.192534:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760711.192542:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760711.192575:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760711.192742:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21659648<>21663744 00000100:00100000:2.0:1789760711.192837:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656832:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760711.214030:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493657088:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.214163:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1789760711.214170:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493656960:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.214204:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493657216:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760711.218687:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493656960:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760711.218716:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493657216:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760711.218727:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493657088:192.168.203.160@tcp:400:kworker.0 00000008:00010000:3.0:1789760711.227355:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760711.227372:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.227382:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.227490:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760711.227518:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493656832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.227619:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5288, 5288] 00000080:00200000:2.0:1789760711.227636:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.227644:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.227658:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21663744 i_blocks 43015 00000020:00200000:2.0:1789760711.227662:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.227671:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.227703:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21663744, count: 4096 00000080:00200000:2.0:1789760711.227710:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21663744, bytes: 4096 00000080:00200000:2.0:1789760711.227722:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.227730:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.227733:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21663744, 21667840) 0 0 00000080:00200000:2.0:1789760711.227737:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.227752:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5289, 5289] 00020000:00200000:2.0:1789760711.227760:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21663744 chunk: [21663744, 21667840) 21667840, 4096 00020000:00200000:2.0:1789760711.227765:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.227783:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5289, 5289] 00000008:00010000:2.0:1789760711.227801:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.227808:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.227833:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21663744, 21667840) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.227848:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21663744=0x14a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.227907:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.227910:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21667840 00000020:00010000:2.0:1789760711.227919:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.227923:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.227926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.227929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.227932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.227941:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.227948:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.227965:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760711.228079:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 21663744<>21667840 00000100:00100000:3.0:1789760711.228298:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.228318:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.259051:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972608 gen 3 00000008:00010000:3.0:1789760711.259117:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760711.259126:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.259138:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.259157:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760711.259186:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.259274:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5289, 5289] 00000080:00200000:2.0:1789760711.259289:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.259295:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.259310:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21667840 i_blocks 43015 00000020:00200000:2.0:1789760711.259313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.259324:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.259357:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21667840, count: 4096 00000080:00200000:2.0:1789760711.259365:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21667840, bytes: 4096 00000080:00200000:2.0:1789760711.259375:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.259383:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.259385:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21667840, 21671936) 0 0 00000080:00200000:2.0:1789760711.259390:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.259438:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5290, 5290] 00020000:00200000:2.0:1789760711.259447:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21667840 chunk: [21667840, 21671936) 21671936, 4096 00020000:00200000:2.0:1789760711.259451:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.259472:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5290, 5290] 00000008:00010000:2.0:1789760711.259490:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.259495:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.259508:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21667840, 21671936) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.259522:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21667840=0x14aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.259589:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.259592:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21671936 00000020:00010000:2.0:1789760711.259601:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.259604:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.259607:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.259622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.259625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.259633:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.259639:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.259655:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760711.259755:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 21667840<>21671936 00000100:00100000:3.0:1789760711.259994:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.293581:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972609 gen 3 00000008:00010000:3.0:1789760711.293646:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760711.293654:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.293666:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.293687:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760711.293716:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657472:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.293871:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5290, 5290] 00000080:00200000:2.0:1789760711.293887:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.293894:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.293910:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21671936 i_blocks 43015 00000020:00200000:2.0:1789760711.293914:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.293924:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.293959:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21671936, count: 4096 00000080:00200000:2.0:1789760711.293966:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21671936, bytes: 4096 00000080:00200000:2.0:1789760711.293976:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.293985:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.293987:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21671936, 21676032) 0 0 00000080:00200000:2.0:1789760711.293991:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.294006:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5291, 5291] 00020000:00200000:2.0:1789760711.294014:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21671936 chunk: [21671936, 21676032) 21676032, 4096 00020000:00200000:2.0:1789760711.294020:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.294040:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5291, 5291] 00000008:00010000:2.0:1789760711.294070:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.294077:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.294088:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21671936, 21676032) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.294102:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21671936=0x14ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.294152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.294155:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21676032 00000020:00010000:2.0:1789760711.294164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.294167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.294170:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.294174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.294176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.294184:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.294191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.294206:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760711.294306:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 21671936<>21676032 00000100:00100000:3.0:1789760711.294568:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.294586:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.331665:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972610 gen 3 00000008:00010000:3.0:1789760711.331718:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760711.331727:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.331738:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.331760:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760711.331791:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.331896:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5291, 5291] 00000080:00200000:2.0:1789760711.331913:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.331920:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.331936:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21676032 i_blocks 43015 00000020:00200000:2.0:1789760711.331940:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.331951:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.331985:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21676032, count: 4096 00000080:00200000:2.0:1789760711.331992:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21676032, bytes: 4096 00000080:00200000:2.0:1789760711.332001:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.332010:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.332012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21676032, 21680128) 0 0 00000080:00200000:2.0:1789760711.332017:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.332045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5292, 5292] 00020000:00200000:2.0:1789760711.332055:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21676032 chunk: [21676032, 21680128) 21680128, 4096 00020000:00200000:2.0:1789760711.332060:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.332080:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5292, 5292] 00000008:00010000:2.0:1789760711.332099:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.332106:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.332118:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21676032, 21680128) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.332132:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21676032=0x14ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.332191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.332193:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21680128 00000020:00010000:2.0:1789760711.332202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.332205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.332209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.332212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.332215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.332222:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.332229:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.332246:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760711.332348:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 21676032<>21680128 00000100:00100000:3.0:1789760711.332535:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657728:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760711.360509:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760711.360524:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.360536:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.360563:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760711.360593:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657728:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.360685:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5292, 5292] 00000080:00200000:2.0:1789760711.360701:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.360707:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.360723:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21680128 i_blocks 43015 00000020:00200000:2.0:1789760711.360727:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.360737:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.360772:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21680128, count: 4096 00000080:00200000:2.0:1789760711.360779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21680128, bytes: 4096 00000080:00200000:2.0:1789760711.360789:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.360797:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.360799:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21680128, 21684224) 0 0 00000080:00200000:2.0:1789760711.360804:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.360832:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5293, 5293] 00020000:00200000:2.0:1789760711.360841:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21680128 chunk: [21680128, 21684224) 21684224, 4096 00020000:00200000:2.0:1789760711.360846:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.360866:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5293, 5293] 00000008:00010000:2.0:1789760711.360885:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.360892:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.360904:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21680128, 21684224) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.360920:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21680128=0x14ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.360975:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.360978:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21684224 00000020:00010000:2.0:1789760711.360987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.360990:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.360993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.360996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.360999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.361006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.361013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.361029:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760711.361139:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21680128<>21684224 00000100:00100000:3.0:1789760711.361200:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493657856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.410885:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972611 gen 3 00000100:00100000:3.0:1789760711.410931:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81180 x1876699493657856/t4294972612(4294972612) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760727 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760711.410981:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760711.410988:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.411000:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.411021:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000080:00200000:2.0:1789760711.411128:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5293, 5293] 00000080:00200000:2.0:1789760711.411147:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.411153:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.411168:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21684224 i_blocks 43015 00000020:00200000:2.0:1789760711.411172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.411182:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.411215:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21684224, count: 4096 00000080:00200000:2.0:1789760711.411223:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21684224, bytes: 4096 00000080:00200000:2.0:1789760711.411234:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.411242:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.411245:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21684224, 21688320) 0 0 00000080:00200000:2.0:1789760711.411250:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.411267:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5294, 5294] 00020000:00200000:2.0:1789760711.411276:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21684224 chunk: [21684224, 21688320) 21688320, 4096 00020000:00200000:2.0:1789760711.411280:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.411298:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5294, 5294] 00000008:00010000:2.0:1789760711.411316:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.411322:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.411336:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21684224, 21688320) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.411350:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21684224=0x14ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.411427:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.411430:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21688320 00000020:00010000:2.0:1789760711.411438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.411442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.411445:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.411449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.411452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.411460:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.411467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760711.411547:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493657856:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760711.411732:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:3.0:1789760711.411838:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21684224<>21688320 00000100:00100000:2.0:1789760711.411899:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.437201:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972612 gen 3 00000100:00100000:3.0:1789760711.437240:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b800 x1876699493657984/t4294972613(4294972613) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760727 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760711.437287:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760711.437294:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.437305:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.437324:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760711.437346:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493657984:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.437782:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5294, 5294] 00000080:00200000:2.0:1789760711.437800:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.437805:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.437824:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21688320 i_blocks 43015 00000020:00200000:2.0:1789760711.437828:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.437838:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.437869:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21688320, count: 4096 00000080:00200000:2.0:1789760711.437876:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21688320, bytes: 4096 00000080:00200000:2.0:1789760711.437885:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.437893:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.437896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21688320, 21692416) 0 0 00000080:00200000:2.0:1789760711.437900:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.437915:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5295, 5295] 00020000:00200000:2.0:1789760711.437922:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21688320 chunk: [21688320, 21692416) 21692416, 4096 00020000:00200000:2.0:1789760711.437927:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.437946:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5295, 5295] 00000008:00010000:2.0:1789760711.437963:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.437969:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.437980:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21688320, 21692416) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.437994:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21688320=0x14af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.438064:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.438067:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21692416 00000020:00010000:2.0:1789760711.438075:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.438079:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.438083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.438086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.438089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.438097:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.438103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.438120:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760711.438214:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 21688320<>21692416 00000100:00100000:3.0:1789760711.438360:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.438380:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658112:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.472657:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972614 gen 3 00000008:00010000:3.0:1789760711.472735:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760711.472743:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.472754:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.473085:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760711.473117:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658112:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760711.473571:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5295, 5295] 00000080:00200000:2.0:1789760711.473590:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760711.473597:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760711.473746:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21692416 i_blocks 43015 00000020:00200000:2.0:1789760711.473751:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760711.473764:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760711.473799:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21692416, count: 4096 00000080:00200000:2.0:1789760711.473806:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21692416, bytes: 4096 00000080:00200000:2.0:1789760711.473819:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760711.473828:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760711.473830:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21692416, 21696512) 0 0 00000080:00200000:2.0:1789760711.473835:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760711.473851:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5296, 5296] 00020000:00200000:2.0:1789760711.473859:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21692416 chunk: [21692416, 21696512) 21696512, 4096 00020000:00200000:2.0:1789760711.473864:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760711.473883:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5296, 5296] 00000008:00010000:2.0:1789760711.473900:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760711.473907:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760711.473920:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21692416, 21696512) append 0 append lockpos 0 00000080:00200000:2.0:1789760711.473933:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21692416=0x14b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760711.474278:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760711.474281:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21696512 00000020:00010000:2.0:1789760711.474290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760711.474293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.474296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760711.474299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760711.474302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760711.474310:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760711.474317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760711.474339:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760711.474721:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 21692416<>21696512 00000100:00100000:3.0:1789760711.474807:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.526769:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972615 gen 3 00000008:00010000:3.0:1789760711.526826:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760711.526835:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.526848:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.526878:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760711.526910:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658240:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760711.688368:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5300, 5300] 00000080:00200000:3.0:1789760711.688385:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760711.688390:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760711.688434:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21712896 i_blocks 43015 00000020:00200000:3.0:1789760711.688438:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760711.688449:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760711.688487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21712896, count: 4096 00000080:00200000:3.0:1789760711.688495:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21712896, bytes: 4096 00000080:00200000:3.0:1789760711.688519:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760711.688529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760711.688531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21712896, 21716992) 0 0 00000080:00200000:3.0:1789760711.688536:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760711.688564:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5301, 5301] 00020000:00200000:3.0:1789760711.688574:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21712896 chunk: [21712896, 21716992) 21716992, 4096 00020000:00200000:3.0:1789760711.688578:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760711.688599:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5301, 5301] 00000008:00010000:3.0:1789760711.688628:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760711.688633:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760711.688645:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21712896, 21716992) append 0 append lockpos 0 00000080:00200000:3.0:1789760711.688660:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21712896=0x14b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760711.688717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760711.688720:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21716992 00000020:00010000:3.0:1789760711.688729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760711.688732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.688735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.688739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760711.688742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760711.688749:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760711.688756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760711.688771:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760711.688874:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21712896<>21716992 00000100:00100000:3.0:1789760711.688918:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.688934:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658880:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760711.721843:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972620 gen 3 00000008:00010000:3.0:1789760711.721908:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760711.721917:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.721926:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.721948:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760711.721978:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493658880:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760711.916722:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760711.985041:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5306, 5306] 00000080:00200000:3.0:1789760711.985060:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760711.985066:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760711.985085:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21737472 i_blocks 43015 00000020:00200000:3.0:1789760711.985089:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760711.985100:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760711.985135:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21737472, count: 4096 00000080:00200000:3.0:1789760711.985142:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21737472, bytes: 4096 00000080:00200000:3.0:1789760711.985152:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760711.985161:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760711.985163:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21737472, 21741568) 0 0 00000080:00200000:3.0:1789760711.985169:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760711.985185:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5307, 5307] 00020000:00200000:3.0:1789760711.985194:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21737472 chunk: [21737472, 21741568) 21741568, 4096 00020000:00200000:3.0:1789760711.985198:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760711.985217:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5307, 5307] 00000008:00010000:3.0:1789760711.985236:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760711.985242:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760711.985254:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21737472, 21741568) append 0 append lockpos 0 00000080:00200000:3.0:1789760711.985269:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21737472=0x14bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760711.985325:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760711.985328:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21741568 00000020:00010000:3.0:1789760711.985336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760711.985340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.985343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760711.985346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760711.985349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760711.985357:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760711.985364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760711.985380:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760711.985534:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 21737472<>21741568 00000100:00100000:2.0:1789760711.985701:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493659648:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760712.051202:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760712.051217:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.051227:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.051250:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760712.051277:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493659648:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760712.111029:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5308, 5308] 00000080:00200000:3.0:1789760712.111053:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760712.111060:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760712.111075:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21745664 i_blocks 43015 00000020:00200000:3.0:1789760712.111080:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760712.111092:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760712.111125:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21745664, count: 4096 00000080:00200000:3.0:1789760712.111132:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21745664, bytes: 4096 00000080:00200000:3.0:1789760712.111143:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760712.111151:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760712.111154:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21745664, 21749760) 0 0 00000080:00200000:3.0:1789760712.111158:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760712.111175:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5309, 5309] 00020000:00200000:3.0:1789760712.111184:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21745664 chunk: [21745664, 21749760) 21749760, 4096 00020000:00200000:3.0:1789760712.111189:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760712.111209:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5309, 5309] 00000008:00010000:3.0:1789760712.111227:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760712.111233:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760712.111245:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21745664, 21749760) append 0 append lockpos 0 00000080:00200000:3.0:1789760712.111259:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21745664=0x14bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760712.111315:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760712.111318:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21749760 00000020:00010000:3.0:1789760712.111326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760712.111330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.111334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760712.111337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760712.111341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760712.111349:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760712.111355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760712.111370:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760712.111521:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 21745664<>21749760 00000100:00100000:3.0:1789760712.111573:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760712.111593:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493659904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.171562:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972628 gen 3 00000008:00010000:3.0:1789760712.171619:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760712.171627:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.171638:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.171698:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760712.171728:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493659904:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.172349:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5309, 5309] 00000080:00200000:2.0:1789760712.172367:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.172374:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.172391:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21749760 i_blocks 43015 00000020:00200000:2.0:1789760712.172429:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.172439:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.172474:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21749760, count: 4096 00000080:00200000:2.0:1789760712.172482:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21749760, bytes: 4096 00000080:00200000:2.0:1789760712.172494:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.172502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.172505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21749760, 21753856) 0 0 00000080:00200000:2.0:1789760712.172509:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.172529:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5310, 5310] 00020000:00200000:2.0:1789760712.172539:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21749760 chunk: [21749760, 21753856) 21753856, 4096 00020000:00200000:2.0:1789760712.172544:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.172574:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5310, 5310] 00000008:00010000:2.0:1789760712.172593:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.172600:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.172723:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21749760, 21753856) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.172739:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21749760=0x14be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.173027:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.173105:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21753856 00000020:00010000:2.0:1789760712.173115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.173119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.173122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.173126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.173129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.173139:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.173146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.173169:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760712.173271:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 21749760<>21753856 00000100:00100000:3.0:1789760712.173358:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493660032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.245631:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972629 gen 3 00000008:00010000:3.0:1789760712.245685:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760712.245694:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.245707:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.245727:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760712.245757:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493660032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.246005:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5310, 5310] 00000080:00200000:2.0:1789760712.246022:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.246037:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.246055:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21753856 i_blocks 43015 00000020:00200000:2.0:1789760712.246058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.246070:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.246105:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21753856, count: 4096 00000080:00200000:2.0:1789760712.246113:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21753856, bytes: 4096 00000080:00200000:2.0:1789760712.246123:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.246133:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.246135:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21753856, 21757952) 0 0 00000080:00200000:2.0:1789760712.246140:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.246156:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5311, 5311] 00020000:00200000:2.0:1789760712.246164:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21753856 chunk: [21753856, 21757952) 21757952, 4096 00020000:00200000:2.0:1789760712.246169:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.246187:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5311, 5311] 00000008:00010000:2.0:1789760712.246205:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.246211:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.246224:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21753856, 21757952) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.246237:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21753856=0x14bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.246293:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.246296:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21757952 00000020:00010000:2.0:1789760712.246304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.246308:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.246311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.246314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.246318:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.246325:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.246332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.246347:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760712.246494:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 21753856<>21757952 00000100:00100000:3.0:1789760712.246832:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660160:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760712.285648:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760712.285665:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.285678:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.285698:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760712.285730:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.285876:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5311, 5311] 00000080:00200000:2.0:1789760712.285892:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.285898:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.285912:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21757952 i_blocks 43015 00000020:00200000:2.0:1789760712.285916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.285927:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.285961:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21757952, count: 4096 00000080:00200000:2.0:1789760712.285967:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21757952, bytes: 4096 00000080:00200000:2.0:1789760712.285978:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.285988:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.285990:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21757952, 21762048) 0 0 00000080:00200000:2.0:1789760712.285995:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.286011:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5312, 5312] 00020000:00200000:2.0:1789760712.286020:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21757952 chunk: [21757952, 21762048) 21762048, 4096 00020000:00200000:2.0:1789760712.286024:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.286056:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5312, 5312] 00000008:00010000:2.0:1789760712.286074:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.286080:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.286090:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21757952, 21762048) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.286104:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21757952=0x14c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.286160:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.286164:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21762048 00000020:00010000:2.0:1789760712.286172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.286176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.286178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.286182:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.286185:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.286193:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.286199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.286214:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760712.286311:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 21757952<>21762048 00000100:00100000:3.0:1789760712.286532:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760712.286560:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.318300:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972631 gen 3 00000008:00010000:3.0:1789760712.318364:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760712.318373:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.318384:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.318432:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760712.318461:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.318620:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5312, 5312] 00000080:00200000:2.0:1789760712.318648:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.318655:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.318672:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21762048 i_blocks 43015 00000020:00200000:2.0:1789760712.318677:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.318689:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.318729:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21762048, count: 4096 00000080:00200000:2.0:1789760712.318737:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21762048, bytes: 4096 00000080:00200000:2.0:1789760712.318747:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.318756:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.318759:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21762048, 21766144) 0 0 00000080:00200000:2.0:1789760712.318763:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.318778:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5313, 5313] 00020000:00200000:2.0:1789760712.318787:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21762048 chunk: [21762048, 21766144) 21766144, 4096 00020000:00200000:2.0:1789760712.318791:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.318810:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5313, 5313] 00000008:00010000:2.0:1789760712.318828:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.318834:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.318846:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21762048, 21766144) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.318991:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21762048=0x14c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.319179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.319183:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21766144 00000020:00010000:2.0:1789760712.319192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.319196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.319199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.319203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.319207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.319215:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.319222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.319241:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760712.319347:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 21762048<>21766144 00000100:00100000:3.0:1789760712.319628:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.358384:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972632 gen 3 00000008:00010000:2.0:1789760712.358841:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:2.0:1789760712.358908:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760712.358921:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760712.358982:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:2.0:1789760712.359011:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493660416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.747000:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5321, 5321] 00000080:00200000:2.0:1789760712.747016:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.747022:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.747038:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21798912 i_blocks 43015 00000020:00200000:2.0:1789760712.747042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.747063:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.747099:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21798912, count: 4096 00000080:00200000:2.0:1789760712.747107:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21798912, bytes: 4096 00000080:00200000:2.0:1789760712.747118:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.747127:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.747129:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21798912, 21803008) 0 0 00000080:00200000:2.0:1789760712.747133:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.747150:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5322, 5322] 00020000:00200000:2.0:1789760712.747158:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21798912 chunk: [21798912, 21803008) 21803008, 4096 00020000:00200000:2.0:1789760712.747162:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.747183:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5322, 5322] 00000008:00010000:2.0:1789760712.747202:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.747207:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.747218:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21798912, 21803008) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.747231:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21798912=0x14ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.747290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.747293:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21803008 00000020:00010000:2.0:1789760712.747302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.747305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.747310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.747313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.747316:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.747324:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.747330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.747347:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760712.747499:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 21798912<>21803008 00000100:00100000:2.0:1789760712.747537:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760712.747566:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493661568:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.783798:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972641 gen 3 00000008:00010000:3.0:1789760712.783858:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760712.783866:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.783880:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.783899:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760712.783929:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493661568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.784298:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5322, 5322] 00000080:00200000:2.0:1789760712.784314:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.784320:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.784336:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21803008 i_blocks 43015 00000020:00200000:2.0:1789760712.784339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.784349:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.784379:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21803008, count: 4096 00000080:00200000:2.0:1789760712.784386:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21803008, bytes: 4096 00000080:00200000:2.0:1789760712.784418:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.784428:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.784430:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21803008, 21807104) 0 0 00000080:00200000:2.0:1789760712.784434:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.784450:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5323, 5323] 00020000:00200000:2.0:1789760712.784459:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21803008 chunk: [21803008, 21807104) 21807104, 4096 00020000:00200000:2.0:1789760712.784462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.784482:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5323, 5323] 00000008:00010000:2.0:1789760712.784499:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.784506:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.784517:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21803008, 21807104) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.784530:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21803008=0x14cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.784601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.784605:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21807104 00000020:00010000:2.0:1789760712.784613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.784616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.784619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.784623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.784625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.784695:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.784703:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.784719:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760712.784820:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 21803008<>21807104 00000100:00100000:3.0:1789760712.785443:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760712.785468:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493661696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.848518:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972642 gen 3 00000008:00010000:3.0:1789760712.848579:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760712.848588:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.848608:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.848630:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760712.848764:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493661696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.848981:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5323, 5323] 00000080:00200000:2.0:1789760712.848999:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.849004:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.849020:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21807104 i_blocks 43015 00000020:00200000:2.0:1789760712.849035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.849149:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.849185:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21807104, count: 4096 00000080:00200000:2.0:1789760712.849192:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21807104, bytes: 4096 00000080:00200000:2.0:1789760712.849276:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.849286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.849288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21807104, 21811200) 0 0 00000080:00200000:2.0:1789760712.849293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.849310:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5324, 5324] 00020000:00200000:2.0:1789760712.849319:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21807104 chunk: [21807104, 21811200) 21811200, 4096 00020000:00200000:2.0:1789760712.849324:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.849344:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5324, 5324] 00000008:00010000:2.0:1789760712.849362:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.849368:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.849380:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21807104, 21811200) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.849423:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21807104=0x14cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.849482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.849486:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21811200 00000020:00010000:2.0:1789760712.849494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.849498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.849501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.849504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.849508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.849516:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.849522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.849539:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760712.849906:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 21807104<>21811200 00000100:00100000:3.0:1789760712.850190:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493661824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.912351:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972643 gen 3 00000008:00010000:3.0:1789760712.912437:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760712.912446:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.912460:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.912481:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760712.912511:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493661824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.912732:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5324, 5324] 00000080:00200000:2.0:1789760712.912754:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.912761:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.912779:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21811200 i_blocks 43015 00000020:00200000:2.0:1789760712.912784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.912797:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.912896:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21811200, count: 4096 00000080:00200000:2.0:1789760712.912904:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21811200, bytes: 4096 00000080:00200000:2.0:1789760712.912915:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.912925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.912927:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21811200, 21815296) 0 0 00000080:00200000:2.0:1789760712.912932:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.912949:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5325, 5325] 00020000:00200000:2.0:1789760712.912958:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21811200 chunk: [21811200, 21815296) 21815296, 4096 00020000:00200000:2.0:1789760712.912963:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.912982:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5325, 5325] 00000008:00010000:2.0:1789760712.913002:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.913008:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.913020:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21811200, 21815296) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.913160:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21811200=0x14cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.913221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.913224:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21815296 00000020:00010000:2.0:1789760712.913233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.913237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.913241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.913244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.913247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.913256:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.913264:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.913283:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760712.913384:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 21811200<>21815296 00000100:00100000:3.0:1789760712.913509:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493661952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760712.979425:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972644 gen 3 00000008:00010000:3.0:1789760712.979478:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760712.979486:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.979498:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.979518:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760712.979556:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493661952:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760712.979663:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5325, 5325] 00000080:00200000:2.0:1789760712.979680:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760712.979686:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760712.979702:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21815296 i_blocks 43015 00000020:00200000:2.0:1789760712.979705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760712.979716:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760712.979751:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21815296, count: 4096 00000080:00200000:2.0:1789760712.979758:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21815296, bytes: 4096 00000080:00200000:2.0:1789760712.979768:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760712.979777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760712.979780:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21815296, 21819392) 0 0 00000080:00200000:2.0:1789760712.979785:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760712.979800:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5326, 5326] 00020000:00200000:2.0:1789760712.979809:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21815296 chunk: [21815296, 21819392) 21819392, 4096 00020000:00200000:2.0:1789760712.979813:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760712.980010:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5326, 5326] 00000008:00010000:2.0:1789760712.980030:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760712.980036:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760712.980120:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21815296, 21819392) append 0 append lockpos 0 00000080:00200000:2.0:1789760712.980136:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21815296=0x14ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760712.980199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760712.980203:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21819392 00000020:00010000:2.0:1789760712.980211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760712.980215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.980218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760712.980221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760712.980224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760712.980232:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760712.980239:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760712.980256:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760712.980355:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 21815296<>21819392 00000100:00100000:3.0:1789760712.981014:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662080:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760713.018987:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760713.019002:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.019014:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.019112:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760713.019147:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.019439:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5326, 5326] 00000080:00200000:2.0:1789760713.019456:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.019463:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.019479:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21819392 i_blocks 43015 00000020:00200000:2.0:1789760713.019483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.019495:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.019527:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21819392, count: 4096 00000080:00200000:2.0:1789760713.019535:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21819392, bytes: 4096 00000080:00200000:2.0:1789760713.019555:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.019565:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.019567:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21819392, 21823488) 0 0 00000080:00200000:2.0:1789760713.019572:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.019589:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5327, 5327] 00020000:00200000:2.0:1789760713.019608:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21819392 chunk: [21819392, 21823488) 21823488, 4096 00020000:00200000:2.0:1789760713.019613:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.019759:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5327, 5327] 00000008:00010000:2.0:1789760713.019780:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.019786:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.019799:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21819392, 21823488) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.019824:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21819392=0x14cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.019881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.019885:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21823488 00000020:00010000:2.0:1789760713.019893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.019897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.019900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.019903:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.019906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.019913:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.019920:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.019936:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760713.020226:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 21819392<>21823488 00000100:00100000:3.0:1789760713.020453:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.020476:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.067467:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972646 gen 3 00000008:00010000:3.0:1789760713.067534:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760713.067553:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.067564:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.067795:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760713.067835:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.067987:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5327, 5327] 00000080:00200000:2.0:1789760713.068003:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.068009:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.068238:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21823488 i_blocks 43015 00000020:00200000:2.0:1789760713.068243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.068254:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.068290:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21823488, count: 4096 00000080:00200000:2.0:1789760713.068299:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21823488, bytes: 4096 00000080:00200000:2.0:1789760713.068310:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.068320:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.068322:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21823488, 21827584) 0 0 00000080:00200000:2.0:1789760713.068327:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.068344:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5328, 5328] 00020000:00200000:2.0:1789760713.068352:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21823488 chunk: [21823488, 21827584) 21827584, 4096 00020000:00200000:2.0:1789760713.068357:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.068377:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5328, 5328] 00000008:00010000:2.0:1789760713.068434:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.068441:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.068538:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21823488, 21827584) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.068565:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21823488=0x14d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.069477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.069483:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21827584 00000020:00010000:2.0:1789760713.069494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.069498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.069502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.069506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.069509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.069519:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.069527:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.069556:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760713.069939:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 21823488<>21827584 00000100:00100000:3.0:1789760713.070039:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662336:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760713.117421:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760713.117439:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.117453:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.117474:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760713.117504:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662336:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.117616:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5328, 5328] 00000080:00200000:2.0:1789760713.117640:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.117646:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.117662:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21827584 i_blocks 43015 00000020:00200000:2.0:1789760713.117666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.117678:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.117713:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21827584, count: 4096 00000080:00200000:2.0:1789760713.117720:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21827584, bytes: 4096 00000080:00200000:2.0:1789760713.117731:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.117740:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.117742:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21827584, 21831680) 0 0 00000080:00200000:2.0:1789760713.117747:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.117763:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5329, 5329] 00020000:00200000:2.0:1789760713.117772:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21827584 chunk: [21827584, 21831680) 21831680, 4096 00020000:00200000:2.0:1789760713.117777:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.117796:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5329, 5329] 00000008:00010000:2.0:1789760713.117814:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.117831:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.117843:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21827584, 21831680) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.117858:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21827584=0x14d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.117916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.117919:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21831680 00000020:00010000:2.0:1789760713.117928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.117932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.117935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.117938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.117941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.117949:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.117956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.117973:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760713.118281:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 21827584<>21831680 00000100:00100000:3.0:1789760713.118354:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493662464:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.234292:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972647 gen 3 00000100:00100000:3.0:1789760713.234337:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83b80 x1876699493662464/t4294972648(4294972648) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.234391:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760713.234428:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.234439:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.234460:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760713.234486:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493662464:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.234585:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5329, 5329] 00000080:00200000:2.0:1789760713.234613:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.234619:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.234635:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21831680 i_blocks 43015 00000020:00200000:2.0:1789760713.234639:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.234649:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.234685:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21831680, count: 4096 00000080:00200000:2.0:1789760713.234692:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21831680, bytes: 4096 00000080:00200000:2.0:1789760713.234701:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.234710:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.234712:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21831680, 21835776) 0 0 00000080:00200000:2.0:1789760713.234717:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.234732:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5330, 5330] 00020000:00200000:2.0:1789760713.234740:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21831680 chunk: [21831680, 21835776) 21835776, 4096 00020000:00200000:2.0:1789760713.234744:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.234762:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5330, 5330] 00000008:00010000:2.0:1789760713.234955:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.234963:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.234976:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21831680, 21835776) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.234991:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21831680=0x14d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.235179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.235183:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21835776 00000020:00010000:2.0:1789760713.235191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.235195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.235197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.235201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.235204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.235213:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.235220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.235238:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760713.235350:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 21831680<>21835776 00000100:00100000:3.0:1789760713.235745:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.322272:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972649 gen 3 00000008:00010000:3.0:1789760713.322374:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760713.322385:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.322415:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.322511:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760713.322550:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493662592:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.323310:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5330, 5330] 00000080:00200000:2.0:1789760713.323329:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.323335:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.323352:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21835776 i_blocks 43015 00000020:00200000:2.0:1789760713.323357:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.323369:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.323442:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21835776, count: 4096 00000080:00200000:2.0:1789760713.323450:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21835776, bytes: 4096 00000080:00200000:2.0:1789760713.323465:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.323476:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.323478:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21835776, 21839872) 0 0 00000080:00200000:2.0:1789760713.323483:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.323503:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5331, 5331] 00020000:00200000:2.0:1789760713.323512:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21835776 chunk: [21835776, 21839872) 21839872, 4096 00020000:00200000:2.0:1789760713.323516:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.323536:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5331, 5331] 00000008:00010000:2.0:1789760713.323566:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.323572:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.323584:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21835776, 21839872) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.323668:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21835776=0x14d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.323735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.323738:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21839872 00000020:00010000:2.0:1789760713.323748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.323752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.323755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.323758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.323761:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.323848:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.323857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.323879:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760713.323989:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21835776<>21839872 00000100:00100000:3.0:1789760713.324341:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493662720:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.437751:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972650 gen 3 00000008:00010000:2.0:1789760713.440886:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760713.440901:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760713.440914:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760713.440946:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760713.440983:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493662720:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.723479:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5335, 5335] 00000080:00200000:2.0:1789760713.723494:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.723500:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.723515:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21856256 i_blocks 43015 00000020:00200000:2.0:1789760713.723519:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.723529:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.723573:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21856256, count: 4096 00000080:00200000:2.0:1789760713.723580:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21856256, bytes: 4096 00000080:00200000:2.0:1789760713.723589:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.723597:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.723600:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21856256, 21860352) 0 0 00000080:00200000:2.0:1789760713.723605:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.723620:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5336, 5336] 00020000:00200000:2.0:1789760713.723628:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21856256 chunk: [21856256, 21860352) 21860352, 4096 00020000:00200000:2.0:1789760713.723633:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.723654:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5336, 5336] 00000008:00010000:2.0:1789760713.723672:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.723678:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.723689:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21856256, 21860352) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.723703:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21856256=0x14d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.723763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.723766:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21860352 00000020:00010000:2.0:1789760713.723775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.723779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.723781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.723785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.723787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.723795:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.723802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.723831:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760713.723931:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21856256<>21860352 00000100:00100000:3.0:1789760713.724009:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663360:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760713.774997:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972655 gen 3 00000008:00010000:2.0:1789760713.775317:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760713.775326:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760713.775338:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760713.775458:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760713.775486:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.824748:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5337, 5337] 00000080:00200000:2.0:1789760713.824767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.824773:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.824791:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21864448 i_blocks 43015 00000020:00200000:2.0:1789760713.824795:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.824807:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.824898:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21864448, count: 4096 00000080:00200000:2.0:1789760713.824907:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21864448, bytes: 4096 00000080:00200000:2.0:1789760713.824919:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.824929:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.824931:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21864448, 21868544) 0 0 00000080:00200000:2.0:1789760713.824936:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.824956:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5338, 5338] 00020000:00200000:2.0:1789760713.824965:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21864448 chunk: [21864448, 21868544) 21868544, 4096 00020000:00200000:2.0:1789760713.824970:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.824990:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5338, 5338] 00000008:00010000:2.0:1789760713.825009:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.825015:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.825103:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21864448, 21868544) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.825119:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21864448=0x14da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.825179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.825182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21868544 00000020:00010000:2.0:1789760713.825191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.825195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.825198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.825202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.825205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.825214:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.825222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.825240:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760713.825342:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21864448<>21868544 00000100:00100000:3.0:1789760713.825461:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493663616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.872636:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972657 gen 3 00000008:00010000:3.0:1789760713.872687:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760713.872696:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.872708:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.872729:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760713.872760:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493663616:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.872931:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5338, 5338] 00000080:00200000:2.0:1789760713.872948:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.872954:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.872969:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21868544 i_blocks 43015 00000020:00200000:2.0:1789760713.872973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.872984:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.873031:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21868544, count: 4096 00000080:00200000:2.0:1789760713.873040:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21868544, bytes: 4096 00000080:00200000:2.0:1789760713.873049:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.873058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.873060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21868544, 21872640) 0 0 00000080:00200000:2.0:1789760713.873065:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.873082:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5339, 5339] 00020000:00200000:2.0:1789760713.873090:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21868544 chunk: [21868544, 21872640) 21872640, 4096 00020000:00200000:2.0:1789760713.873095:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.873114:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5339, 5339] 00000008:00010000:2.0:1789760713.873134:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.873140:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.873153:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21868544, 21872640) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.873167:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21868544=0x14db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.873237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.873240:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21872640 00000020:00010000:2.0:1789760713.873250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.873254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.873257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.873261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.873264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.873272:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.873278:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.873296:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760713.873435:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 21868544<>21872640 00000100:00100000:3.0:1789760713.873901:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663744:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760713.923281:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972658 gen 3 00000008:00010000:3.0:1789760713.923334:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760713.923344:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.923356:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.923377:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760713.923448:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.923606:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5339, 5339] 00000080:00200000:2.0:1789760713.923624:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.923793:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.923810:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21872640 i_blocks 43015 00000020:00200000:2.0:1789760713.923814:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.923838:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.923875:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21872640, count: 4096 00000080:00200000:2.0:1789760713.923883:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21872640, bytes: 4096 00000080:00200000:2.0:1789760713.923894:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.923904:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.923906:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21872640, 21876736) 0 0 00000080:00200000:2.0:1789760713.923911:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.923926:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5340, 5340] 00020000:00200000:2.0:1789760713.923934:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21872640 chunk: [21872640, 21876736) 21876736, 4096 00020000:00200000:2.0:1789760713.923939:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.923959:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5340, 5340] 00000008:00010000:2.0:1789760713.923978:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.923985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.923998:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21872640, 21876736) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.924014:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21872640=0x14dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.924208:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.924212:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21876736 00000020:00010000:2.0:1789760713.924221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.924224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.924228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.924232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.924235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.924243:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.924250:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.924267:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760713.924373:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 21872640<>21876736 00000100:00100000:3.0:1789760713.924502:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.924524:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663872:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760713.989929:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760713.989948:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.989961:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.989982:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760713.990013:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493663872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760713.990114:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5340, 5340] 00000080:00200000:2.0:1789760713.990131:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760713.990136:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760713.990151:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21876736 i_blocks 43015 00000020:00200000:2.0:1789760713.990155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760713.990165:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760713.990197:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21876736, count: 4096 00000080:00200000:2.0:1789760713.990204:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21876736, bytes: 4096 00000080:00200000:2.0:1789760713.990214:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760713.990222:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760713.990224:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21876736, 21880832) 0 0 00000080:00200000:2.0:1789760713.990228:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760713.990243:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5341, 5341] 00020000:00200000:2.0:1789760713.990251:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21876736 chunk: [21876736, 21880832) 21880832, 4096 00020000:00200000:2.0:1789760713.990255:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760713.990273:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5341, 5341] 00000008:00010000:2.0:1789760713.990290:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760713.990295:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760713.990307:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21876736, 21880832) append 0 append lockpos 0 00000080:00200000:2.0:1789760713.990319:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21876736=0x14dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760713.990373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760713.990376:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21880832 00000020:00010000:2.0:1789760713.990385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760713.990389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.990391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760713.990435:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760713.990438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760713.990445:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760713.990452:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760713.990467:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760713.990569:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 21876736<>21880832 00000100:00100000:3.0:1789760713.991419:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493664000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.049374:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972660 gen 3 00000008:00010000:3.0:1789760714.049470:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760714.049479:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.049490:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.049510:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760714.049547:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493664000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.049946:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5341, 5341] 00000080:00200000:2.0:1789760714.049964:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.049969:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.049985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21880832 i_blocks 43015 00000020:00200000:2.0:1789760714.049989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.050000:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.050046:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21880832, count: 4096 00000080:00200000:2.0:1789760714.050053:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21880832, bytes: 4096 00000080:00200000:2.0:1789760714.050065:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.050075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.050077:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21880832, 21884928) 0 0 00000080:00200000:2.0:1789760714.050082:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.050099:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5342, 5342] 00020000:00200000:2.0:1789760714.050108:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21880832 chunk: [21880832, 21884928) 21884928, 4096 00020000:00200000:2.0:1789760714.050113:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.050134:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5342, 5342] 00000008:00010000:2.0:1789760714.050155:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.050161:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.050175:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21880832, 21884928) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.050191:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21880832=0x14de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.050477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.050481:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21884928 00000020:00010000:2.0:1789760714.050490:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.050493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.050496:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.050500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.050502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.050521:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.050529:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.050556:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760714.051114:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 21880832<>21884928 00000100:00100000:3.0:1789760714.051210:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493664128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.096962:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972661 gen 3 00000008:00010000:3.0:1789760714.097025:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760714.097033:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.097044:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.097066:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760714.097093:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493664128:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760714.279503:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5346, 5346] 00000080:00200000:3.0:1789760714.279519:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.279525:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.279557:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21901312 i_blocks 43015 00000020:00200000:3.0:1789760714.279562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.279573:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.279618:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21901312, count: 4096 00000080:00200000:3.0:1789760714.279742:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21901312, bytes: 4096 00000080:00200000:3.0:1789760714.279756:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.279765:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.279768:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21901312, 21905408) 0 0 00000080:00200000:3.0:1789760714.279773:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.279789:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5347, 5347] 00020000:00200000:3.0:1789760714.279798:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21901312 chunk: [21901312, 21905408) 21905408, 4096 00020000:00200000:3.0:1789760714.279802:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.279833:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5347, 5347] 00000008:00010000:3.0:1789760714.279851:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.279859:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760714.279872:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21901312, 21905408) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.279885:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21901312=0x14e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.279947:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.279949:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21905408 00000020:00010000:3.0:1789760714.279958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.279961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.279964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.279968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.279971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.279979:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.279986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.280004:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760714.280186:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 21901312<>21905408 00000100:00100000:2.0:1789760714.280267:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493664768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.319466:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972666 gen 3 00000008:00010000:3.0:1789760714.319565:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760714.319574:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.319585:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.319616:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760714.319641:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493664768:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760714.360013:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5348, 5348] 00000080:00200000:3.0:1789760714.360146:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.360153:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.360170:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21909504 i_blocks 43015 00000020:00200000:3.0:1789760714.360174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.360184:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.360218:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21909504, count: 4096 00000080:00200000:3.0:1789760714.360226:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21909504, bytes: 4096 00000080:00200000:3.0:1789760714.360238:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.360246:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.360249:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21909504, 21913600) 0 0 00000080:00200000:3.0:1789760714.360253:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.360270:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5349, 5349] 00020000:00200000:3.0:1789760714.360279:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21909504 chunk: [21909504, 21913600) 21913600, 4096 00020000:00200000:3.0:1789760714.360283:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.360303:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5349, 5349] 00000008:00010000:3.0:1789760714.360322:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.360328:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760714.360341:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21909504, 21913600) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.360355:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21909504=0x14e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.360518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.360522:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21913600 00000020:00010000:3.0:1789760714.360530:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.360545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.360548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.360552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.360556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.360563:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.360635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.360653:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760714.360759:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 21909504<>21913600 00000100:00100000:2.0:1789760714.360836:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.409281:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972667 gen 3 00000100:00100000:3.0:1789760714.409328:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148000 x1876699493665024/t4294972668(4294972668) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760730 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760714.409379:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760714.409387:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.409427:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.409449:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760714.409473:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.409636:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5349, 5349] 00000080:00200000:2.0:1789760714.409653:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.409660:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.409676:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21913600 i_blocks 43015 00000020:00200000:2.0:1789760714.409680:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.409690:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.409722:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21913600, count: 4096 00000080:00200000:2.0:1789760714.409729:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21913600, bytes: 4096 00000080:00200000:2.0:1789760714.409739:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.409749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.409751:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21913600, 21917696) 0 0 00000080:00200000:2.0:1789760714.409756:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.409773:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5350, 5350] 00020000:00200000:2.0:1789760714.409781:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21913600 chunk: [21913600, 21917696) 21917696, 4096 00020000:00200000:2.0:1789760714.409785:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.409804:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5350, 5350] 00000008:00010000:2.0:1789760714.409823:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.409829:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.409841:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21913600, 21917696) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.409857:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21913600=0x14e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.409916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.409918:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21917696 00000020:00010000:2.0:1789760714.409926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.409930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.409933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.409936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.409939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.409948:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.409955:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.409970:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760714.410298:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 21913600<>21917696 00000100:00100000:3.0:1789760714.410359:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.475694:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972669 gen 3 00000008:00010000:3.0:1789760714.475761:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760714.475769:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.475779:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.475797:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760714.475829:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.475998:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5350, 5350] 00000080:00200000:2.0:1789760714.476119:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.476126:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.476141:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21917696 i_blocks 43015 00000020:00200000:2.0:1789760714.476146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.476157:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.476192:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21917696, count: 4096 00000080:00200000:2.0:1789760714.476200:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21917696, bytes: 4096 00000080:00200000:2.0:1789760714.476209:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.476217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.476219:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21917696, 21921792) 0 0 00000080:00200000:2.0:1789760714.476223:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.476238:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5351, 5351] 00020000:00200000:2.0:1789760714.476246:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21917696 chunk: [21917696, 21921792) 21921792, 4096 00020000:00200000:2.0:1789760714.476250:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.476268:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5351, 5351] 00000008:00010000:2.0:1789760714.476288:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.476294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.476306:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21917696, 21921792) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.476321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21917696=0x14e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.476379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.476382:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21921792 00000020:00010000:2.0:1789760714.476391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.476740:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.476744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.476747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.476750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.476761:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.476768:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.476785:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760714.476890:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 21917696<>21921792 00000100:00100000:3.0:1789760714.476962:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665280:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.532629:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972670 gen 3 00000008:00010000:3.0:1789760714.532680:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760714.532689:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.532701:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.532724:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760714.532756:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.535079:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5351, 5351] 00000080:00200000:2.0:1789760714.535111:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.535118:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.535137:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21921792 i_blocks 43015 00000020:00200000:2.0:1789760714.535144:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.535164:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.535211:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21921792, count: 4096 00000080:00200000:2.0:1789760714.535222:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21921792, bytes: 4096 00000080:00200000:2.0:1789760714.535241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.535250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.535252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21921792, 21925888) 0 0 00000080:00200000:2.0:1789760714.535257:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.535274:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5352, 5352] 00020000:00200000:2.0:1789760714.535286:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21921792 chunk: [21921792, 21925888) 21925888, 4096 00020000:00200000:2.0:1789760714.535291:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.535315:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5352, 5352] 00000008:00010000:2.0:1789760714.535338:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.535347:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.535363:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21921792, 21925888) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.535379:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21921792=0x14e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.535479:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.535482:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21925888 00000020:00010000:2.0:1789760714.535491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.535495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.535498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.535502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.535506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.535515:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.535524:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.535559:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760714.535920:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 21921792<>21925888 00000100:00100000:3.0:1789760714.537529:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665408:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760714.597538:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760714.597556:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.597579:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.597597:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760714.597627:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.597770:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5352, 5352] 00000080:00200000:2.0:1789760714.597788:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.597796:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.597813:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21925888 i_blocks 43015 00000020:00200000:2.0:1789760714.597818:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.597829:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.597868:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21925888, count: 4096 00000080:00200000:2.0:1789760714.597875:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21925888, bytes: 4096 00000080:00200000:2.0:1789760714.597885:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.597894:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.597896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21925888, 21929984) 0 0 00000080:00200000:2.0:1789760714.597900:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.597917:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5353, 5353] 00020000:00200000:2.0:1789760714.597924:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21925888 chunk: [21925888, 21929984) 21929984, 4096 00020000:00200000:2.0:1789760714.597928:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.597951:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5353, 5353] 00000008:00010000:2.0:1789760714.597970:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.597985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.597998:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21925888, 21929984) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.598013:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21925888=0x14e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.598074:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.598077:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21929984 00000020:00010000:2.0:1789760714.598086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.598090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.598093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.598097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.598100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.598109:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.598116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.598133:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760714.598246:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 21925888<>21929984 00000100:00100000:3.0:1789760714.598485:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665536:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.652161:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972671 gen 3 00000100:00100000:3.0:1789760714.652205:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f82680 x1876699493665536/t4294972672(4294972672) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760730 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760714.652256:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760714.652263:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.652274:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.652336:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760714.652365:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493665536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760714.652544:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5353, 5353] 00000080:00200000:2.0:1789760714.652561:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760714.652707:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760714.652724:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21929984 i_blocks 43015 00000020:00200000:2.0:1789760714.652728:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760714.652740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760714.652777:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21929984, count: 4096 00000080:00200000:2.0:1789760714.652785:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21929984, bytes: 4096 00000080:00200000:2.0:1789760714.652796:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760714.652805:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760714.652807:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21929984, 21934080) 0 0 00000080:00200000:2.0:1789760714.652812:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760714.652830:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5354, 5354] 00020000:00200000:2.0:1789760714.652839:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21929984 chunk: [21929984, 21934080) 21934080, 4096 00020000:00200000:2.0:1789760714.652843:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760714.652863:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5354, 5354] 00000008:00010000:2.0:1789760714.652882:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760714.652887:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760714.652900:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21929984, 21934080) append 0 append lockpos 0 00000080:00200000:2.0:1789760714.652914:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21929984=0x14ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760714.653047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760714.653051:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21934080 00000020:00010000:2.0:1789760714.653060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760714.653064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.653067:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760714.653071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760714.653074:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760714.653082:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760714.653089:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760714.653108:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760714.653210:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 21929984<>21934080 00000100:00100000:3.0:1789760714.653283:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760714.720271:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972673 gen 3 00000008:00010000:3.0:1789760714.720342:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760714.720351:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.720363:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.720389:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760714.720666:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493665664:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760714.934468:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5357, 5357] 00000080:00200000:3.0:1789760714.934485:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.934492:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.934509:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21946368 i_blocks 43015 00000020:00200000:3.0:1789760714.934513:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.934524:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.934575:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21946368, count: 4096 00000080:00200000:3.0:1789760714.934583:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21946368, bytes: 4096 00000080:00200000:3.0:1789760714.934593:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.934603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.934605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21946368, 21950464) 0 0 00000080:00200000:3.0:1789760714.934610:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.934626:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5358, 5358] 00020000:00200000:3.0:1789760714.934635:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21946368 chunk: [21946368, 21950464) 21950464, 4096 00020000:00200000:3.0:1789760714.934639:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.934659:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5358, 5358] 00000008:00010000:3.0:1789760714.934677:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.934683:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760714.934694:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21946368, 21950464) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.934709:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21946368=0x14ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.934771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.934773:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21950464 00000020:00010000:3.0:1789760714.934781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.934785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.934788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.934792:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.934795:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.934803:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.934810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.934826:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760714.934926:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 21946368<>21950464 00000100:00100000:3.0:1789760714.934977:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760714.934995:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493666176:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760714.988850:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760714.994136:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972677 gen 3 00000008:00010000:3.0:1789760714.994187:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760714.994195:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.994206:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.994229:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760714.994260:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493666176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.038479:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5359, 5359] 00000080:00200000:2.0:1789760715.038498:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.038504:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.038521:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21954560 i_blocks 43015 00000020:00200000:2.0:1789760715.038526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.038547:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.038590:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21954560, count: 4096 00000080:00200000:2.0:1789760715.038597:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21954560, bytes: 4096 00000080:00200000:2.0:1789760715.038608:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.038625:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.038628:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21954560, 21958656) 0 0 00000080:00200000:2.0:1789760715.038632:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.038648:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5360, 5360] 00020000:00200000:2.0:1789760715.038655:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21954560 chunk: [21954560, 21958656) 21958656, 4096 00020000:00200000:2.0:1789760715.038660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.038680:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5360, 5360] 00000008:00010000:2.0:1789760715.038697:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.038703:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.038715:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21954560, 21958656) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.038730:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21954560=0x14f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.038790:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.038793:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21958656 00000020:00010000:2.0:1789760715.038801:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.038804:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.038806:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.038809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.038813:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.038821:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.038828:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.038845:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760715.038948:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 21954560<>21958656 00000100:00100000:3.0:1789760715.039194:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493666432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.122954:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972679 gen 3 00000008:00010000:3.0:1789760715.123350:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760715.123361:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.123374:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.123477:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760715.123506:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493666432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.190594:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5361, 5361] 00000080:00200000:2.0:1789760715.190611:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.190692:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.190708:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21962752 i_blocks 43015 00000020:00200000:2.0:1789760715.190714:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.190725:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.190828:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21962752, count: 4096 00000080:00200000:2.0:1789760715.190837:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21962752, bytes: 4096 00000080:00200000:2.0:1789760715.190849:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.190858:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.190861:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21962752, 21966848) 0 0 00000080:00200000:2.0:1789760715.190866:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.190882:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5362, 5362] 00020000:00200000:2.0:1789760715.190891:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21962752 chunk: [21962752, 21966848) 21966848, 4096 00020000:00200000:2.0:1789760715.190895:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.190915:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5362, 5362] 00000008:00010000:2.0:1789760715.190932:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.190938:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.190951:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21962752, 21966848) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.190965:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21962752=0x14f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.191206:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.191210:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21966848 00000020:00010000:2.0:1789760715.191219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.191224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.191227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.191231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.191235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.191244:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.191251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.191270:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760715.191368:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 21962752<>21966848 00000100:00100000:3.0:1789760715.191612:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666688:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.258022:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972681 gen 3 00000008:00010000:3.0:1789760715.258188:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760715.258197:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.258209:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.258231:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760715.258263:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666688:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.258922:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5362, 5362] 00000080:00200000:2.0:1789760715.258940:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.258947:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.258965:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21966848 i_blocks 43015 00000020:00200000:2.0:1789760715.258971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.258983:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.259161:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21966848, count: 4096 00000080:00200000:2.0:1789760715.259169:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21966848, bytes: 4096 00000080:00200000:2.0:1789760715.259182:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.259192:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.259194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21966848, 21970944) 0 0 00000080:00200000:2.0:1789760715.259198:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.259217:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5363, 5363] 00020000:00200000:2.0:1789760715.259226:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21966848 chunk: [21966848, 21970944) 21970944, 4096 00020000:00200000:2.0:1789760715.259231:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.259252:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5363, 5363] 00000008:00010000:2.0:1789760715.259271:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.259277:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.259290:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21966848, 21970944) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.259305:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21966848=0x14f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.259369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.259372:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21970944 00000020:00010000:2.0:1789760715.259380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.259384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.259387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.259391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.259441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.259451:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.259457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.259476:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760715.259610:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 21966848<>21970944 00000100:00100000:3.0:1789760715.260347:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760715.260374:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666816:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760715.355863:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760715.355880:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.355895:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.355914:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760715.355945:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666816:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.357138:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5363, 5363] 00000080:00200000:2.0:1789760715.357161:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.357168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.357186:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21970944 i_blocks 43015 00000020:00200000:2.0:1789760715.357192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.357205:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.357242:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21970944, count: 4096 00000080:00200000:2.0:1789760715.357250:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21970944, bytes: 4096 00000080:00200000:2.0:1789760715.357264:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.357273:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.357275:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21970944, 21975040) 0 0 00000080:00200000:2.0:1789760715.357281:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.357297:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5364, 5364] 00020000:00200000:2.0:1789760715.357305:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21970944 chunk: [21970944, 21975040) 21975040, 4096 00020000:00200000:2.0:1789760715.357310:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.357333:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5364, 5364] 00000008:00010000:2.0:1789760715.357353:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.357360:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.357375:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21970944, 21975040) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.357423:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21970944=0x14f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.357485:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.357488:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21975040 00000020:00010000:2.0:1789760715.357497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.357501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.357505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.357509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.357512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.357521:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.357540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.357561:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760715.360071:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 21970944<>21975040 00000100:00100000:3.0:1789760715.360361:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.416493:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972683 gen 3 00000008:00010000:3.0:1789760715.416575:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760715.416589:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.416603:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.416620:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760715.416648:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493666944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.416849:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5364, 5364] 00000080:00200000:2.0:1789760715.416865:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.416872:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.416888:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21975040 i_blocks 43015 00000020:00200000:2.0:1789760715.416893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.416904:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.416939:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21975040, count: 4096 00000080:00200000:2.0:1789760715.416945:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21975040, bytes: 4096 00000080:00200000:2.0:1789760715.416956:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.416965:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.416967:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21975040, 21979136) 0 0 00000080:00200000:2.0:1789760715.416972:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.416987:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5365, 5365] 00020000:00200000:2.0:1789760715.416995:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21975040 chunk: [21975040, 21979136) 21979136, 4096 00020000:00200000:2.0:1789760715.416999:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.417030:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5365, 5365] 00000008:00010000:2.0:1789760715.417049:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.417055:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.417068:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21975040, 21979136) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.417082:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21975040=0x14f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.417138:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.417141:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21979136 00000020:00010000:2.0:1789760715.417149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.417154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.417157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.417161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.417164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.417172:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.417178:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.417196:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760715.417300:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 21975040<>21979136 00000100:00100000:3.0:1789760715.417540:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.492724:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972684 gen 3 00000008:00010000:2.0:1789760715.492896:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:2.0:1789760715.492906:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760715.492917:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760715.493001:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:2.0:1789760715.493028:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.697914:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5367, 5367] 00000080:00200000:2.0:1789760715.697934:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.697940:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.697956:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21987328 i_blocks 43015 00000020:00200000:2.0:1789760715.697960:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.697974:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.698006:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21987328, count: 4096 00000080:00200000:2.0:1789760715.698014:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21987328, bytes: 4096 00000080:00200000:2.0:1789760715.698028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.698110:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.698113:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21987328, 21991424) 0 0 00000080:00200000:2.0:1789760715.698117:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.698136:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5368, 5368] 00020000:00200000:2.0:1789760715.698144:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21987328 chunk: [21987328, 21991424) 21991424, 4096 00020000:00200000:2.0:1789760715.698149:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.698170:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5368, 5368] 00000008:00010000:2.0:1789760715.698259:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.698265:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.698278:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21987328, 21991424) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.698293:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21987328=0x14f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.698361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.698364:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21991424 00000020:00010000:2.0:1789760715.698373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.698377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.698380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.698384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.698386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.698435:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.698443:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.698463:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760715.698588:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 21987328<>21991424 00000100:00100000:3.0:1789760715.698890:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667456:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.773858:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972687 gen 3 00000008:00010000:3.0:1789760715.773919:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760715.773930:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.773943:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.773965:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760715.773999:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.774264:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5368, 5368] 00000080:00200000:2.0:1789760715.774282:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.774288:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.774306:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21991424 i_blocks 43015 00000020:00200000:2.0:1789760715.774310:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.774322:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.774361:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21991424, count: 4096 00000080:00200000:2.0:1789760715.774369:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21991424, bytes: 4096 00000080:00200000:2.0:1789760715.774381:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.774391:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.774426:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21991424, 21995520) 0 0 00000080:00200000:2.0:1789760715.774432:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.774452:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5369, 5369] 00020000:00200000:2.0:1789760715.774462:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21991424 chunk: [21991424, 21995520) 21995520, 4096 00020000:00200000:2.0:1789760715.774467:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.774489:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5369, 5369] 00000008:00010000:2.0:1789760715.774507:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.774513:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.774534:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21991424, 21995520) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.774549:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21991424=0x14f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.774617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.774620:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21995520 00000020:00010000:2.0:1789760715.774629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.774633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.774636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.774640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.774643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.774651:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.774659:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.774678:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760715.774789:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 21991424<>21995520 00000100:00100000:2.0:1789760715.774953:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667584:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.857168:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972688 gen 3 00000008:00010000:3.0:1789760715.857227:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760715.857236:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.857249:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.857269:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760715.857299:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.857432:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5369, 5369] 00000080:00200000:2.0:1789760715.857450:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.857456:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.857471:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21995520 i_blocks 43015 00000020:00200000:2.0:1789760715.857475:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.857486:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.857529:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21995520, count: 4096 00000080:00200000:2.0:1789760715.857538:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21995520, bytes: 4096 00000080:00200000:2.0:1789760715.857549:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.857559:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.857561:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21995520, 21999616) 0 0 00000080:00200000:2.0:1789760715.857566:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.857582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5370, 5370] 00020000:00200000:2.0:1789760715.857592:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21995520 chunk: [21995520, 21999616) 21999616, 4096 00020000:00200000:2.0:1789760715.857596:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.857615:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5370, 5370] 00000008:00010000:2.0:1789760715.857635:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.857642:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.857654:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21995520, 21999616) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.857667:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21995520=0x14fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.857729:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.857732:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21999616 00000020:00010000:2.0:1789760715.857752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.857756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.857759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.857763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.857765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.857774:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.857781:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.857799:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760715.857904:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 21995520<>21999616 00000100:00100000:3.0:1789760715.858128:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.919972:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972689 gen 3 00000008:00010000:3.0:1789760715.920033:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760715.920042:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.920055:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.920076:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760715.920106:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.920254:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5370, 5370] 00000080:00200000:2.0:1789760715.920269:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.920275:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.920291:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21999616 i_blocks 43015 00000020:00200000:2.0:1789760715.920295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.920306:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.920341:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21999616, count: 4096 00000080:00200000:2.0:1789760715.920348:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21999616, bytes: 4096 00000080:00200000:2.0:1789760715.920358:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.920366:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.920369:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21999616, 22003712) 0 0 00000080:00200000:2.0:1789760715.920373:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.920389:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5371, 5371] 00020000:00200000:2.0:1789760715.920427:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21999616 chunk: [21999616, 22003712) 22003712, 4096 00020000:00200000:2.0:1789760715.920432:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.920452:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5371, 5371] 00000008:00010000:2.0:1789760715.920470:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.920476:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.920489:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21999616, 22003712) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.920503:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=21999616=0x14fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.920572:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.920575:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22003712 00000020:00010000:2.0:1789760715.920584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.920587:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.920591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.920594:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.920596:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.920605:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.920612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.920627:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760715.920731:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 21999616<>22003712 00000100:00100000:3.0:1789760715.920830:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760715.969121:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972690 gen 3 00000008:00010000:3.0:1789760715.969177:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760715.969187:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.969200:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.969280:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760715.969310:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493667840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760715.969418:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5371, 5371] 00000080:00200000:2.0:1789760715.969436:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760715.969442:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760715.969459:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22003712 i_blocks 43015 00000020:00200000:2.0:1789760715.969463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760715.969474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760715.969510:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22003712, count: 4096 00000080:00200000:2.0:1789760715.969517:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22003712, bytes: 4096 00000080:00200000:2.0:1789760715.969609:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760715.969620:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760715.969622:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22003712, 22007808) 0 0 00000080:00200000:2.0:1789760715.969627:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760715.969643:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5372, 5372] 00020000:00200000:2.0:1789760715.969651:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22003712 chunk: [22003712, 22007808) 22007808, 4096 00020000:00200000:2.0:1789760715.969656:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760715.969676:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5372, 5372] 00000008:00010000:2.0:1789760715.969694:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760715.969700:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760715.969712:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22003712, 22007808) append 0 append lockpos 0 00000080:00200000:2.0:1789760715.969726:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22003712=0x14fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760715.970361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760715.970366:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22007808 00000020:00010000:2.0:1789760715.970377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760715.970380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.970384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760715.970387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760715.970390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760715.970446:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760715.970453:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760715.970477:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760715.970765:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22003712<>22007808 00000100:00100000:3.0:1789760715.970858:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667968:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760716.045091:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760716.045107:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.045119:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.045139:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760716.045170:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493667968:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.045315:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5372, 5372] 00000080:00200000:2.0:1789760716.045331:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.045337:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.045352:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22007808 i_blocks 43015 00000020:00200000:2.0:1789760716.045356:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.045367:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.045428:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22007808, count: 4096 00000080:00200000:2.0:1789760716.045437:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22007808, bytes: 4096 00000080:00200000:2.0:1789760716.045446:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.045456:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.045458:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22007808, 22011904) 0 0 00000080:00200000:2.0:1789760716.045463:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.045479:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5373, 5373] 00020000:00200000:2.0:1789760716.045488:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22007808 chunk: [22007808, 22011904) 22011904, 4096 00020000:00200000:2.0:1789760716.045493:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.045511:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5373, 5373] 00000008:00010000:2.0:1789760716.045540:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.045547:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.045559:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22007808, 22011904) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.045583:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22007808=0x14fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.045638:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.045641:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22011904 00000020:00010000:2.0:1789760716.045649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.045653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.045656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.045659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.045662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.045670:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.045676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.045690:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760716.045794:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22007808<>22011904 00000100:00100000:2.0:1789760716.045843:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.113294:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972692 gen 3 00000008:00010000:3.0:1789760716.113360:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760716.113368:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.113380:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.113426:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760716.113456:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668096:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.113789:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5373, 5373] 00000080:00200000:2.0:1789760716.113805:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.113811:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.113828:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22011904 i_blocks 43015 00000020:00200000:2.0:1789760716.113831:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.113841:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.113876:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22011904, count: 4096 00000080:00200000:2.0:1789760716.113884:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22011904, bytes: 4096 00000080:00200000:2.0:1789760716.113895:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.113905:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.113907:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22011904, 22016000) 0 0 00000080:00200000:2.0:1789760716.113912:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.113930:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5374, 5374] 00020000:00200000:2.0:1789760716.113939:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22011904 chunk: [22011904, 22016000) 22016000, 4096 00020000:00200000:2.0:1789760716.113944:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.113964:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5374, 5374] 00000008:00010000:2.0:1789760716.113983:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.113990:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.114012:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22011904, 22016000) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.114026:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22011904=0x14fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.114087:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.114090:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22016000 00000020:00010000:2.0:1789760716.114099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.114103:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.114106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.114110:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.114114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.114122:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.114129:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.114147:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760716.114252:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 22011904<>22016000 00000100:00100000:3.0:1789760716.115013:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668224:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760716.174213:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760716.174231:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.174245:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.174271:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760716.174305:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668224:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.174502:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5374, 5374] 00000080:00200000:2.0:1789760716.174530:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.174538:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.174553:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22016000 i_blocks 43015 00000020:00200000:2.0:1789760716.174567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.174693:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.174728:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22016000, count: 4096 00000080:00200000:2.0:1789760716.174736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22016000, bytes: 4096 00000080:00200000:2.0:1789760716.174751:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.174760:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.174762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22016000, 22020096) 0 0 00000080:00200000:2.0:1789760716.174767:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.174784:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5375, 5375] 00020000:00200000:2.0:1789760716.174793:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22016000 chunk: [22016000, 22020096) 22020096, 4096 00020000:00200000:2.0:1789760716.174798:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.174949:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5375, 5375] 00000008:00010000:2.0:1789760716.175097:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.175105:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.175119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22016000, 22020096) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.175134:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22016000=0x14ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.175196:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.175199:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22020096 00000020:00010000:2.0:1789760716.175209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.175212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.175216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.175220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.175223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.175232:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.175240:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.175259:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760716.175364:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 22016000<>22020096 00000100:00100000:3.0:1789760716.175922:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668352:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.219241:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972694 gen 3 00000008:00010000:3.0:1789760716.219312:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760716.219321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.219333:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.219353:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760716.219384:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668352:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.219582:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5375, 5375] 00000080:00200000:2.0:1789760716.219599:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.219605:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.219620:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22020096 i_blocks 43015 00000020:00200000:2.0:1789760716.219624:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.219634:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.219669:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22020096, count: 4096 00000080:00200000:2.0:1789760716.219677:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22020096, bytes: 4096 00000080:00200000:2.0:1789760716.219687:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.219696:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.219698:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22020096, 22024192) 0 0 00000080:00200000:2.0:1789760716.219703:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.219717:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5376, 5376] 00020000:00200000:2.0:1789760716.219726:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22020096 chunk: [22020096, 22024192) 22024192, 4096 00020000:00200000:2.0:1789760716.219731:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.219750:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5376, 5376] 00000008:00010000:2.0:1789760716.219770:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.219776:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.219787:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22020096, 22024192) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.219801:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22020096=0x1500000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.219872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.219875:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22024192 00000020:00010000:2.0:1789760716.219883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.219887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.219890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.219894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.219897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.219904:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.219912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.219928:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760716.221754:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22020096<>22024192 00000100:00100000:3.0:1789760716.221870:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.276154:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972695 gen 3 00000008:00010000:3.0:1789760716.276209:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760716.276218:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.276230:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.276253:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760716.276283:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668480:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.277641:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5376, 5376] 00000080:00200000:2.0:1789760716.277662:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.277668:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.277684:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22024192 i_blocks 43015 00000020:00200000:2.0:1789760716.277689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.277703:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.277735:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22024192, count: 4096 00000080:00200000:2.0:1789760716.277743:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22024192, bytes: 4096 00000080:00200000:2.0:1789760716.277757:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.277766:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.277768:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22024192, 22028288) 0 0 00000080:00200000:2.0:1789760716.277773:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.277788:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5377, 5377] 00020000:00200000:2.0:1789760716.277797:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22024192 chunk: [22024192, 22028288) 22028288, 4096 00020000:00200000:2.0:1789760716.277802:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.277820:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5377, 5377] 00000008:00010000:2.0:1789760716.277839:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.277845:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.277857:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22024192, 22028288) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.277872:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22024192=0x1501000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.277938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.277941:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22028288 00000020:00010000:2.0:1789760716.277960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.278164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.278169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.278173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.278176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.278186:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.278194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.278216:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760716.278325:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22024192<>22028288 00000100:00100000:2.0:1789760716.278435:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668608:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760716.327513:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760716.327544:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.327559:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.327584:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760716.327613:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493668608:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.327707:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5377, 5377] 00000080:00200000:2.0:1789760716.327723:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.327729:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.327744:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22028288 i_blocks 45063 00000020:00200000:2.0:1789760716.327748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.327759:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.327793:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22028288, count: 4096 00000080:00200000:2.0:1789760716.327801:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22028288, bytes: 4096 00000080:00200000:2.0:1789760716.327811:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.327830:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.327832:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22028288, 22032384) 0 0 00000080:00200000:2.0:1789760716.327836:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.327852:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5378, 5378] 00020000:00200000:2.0:1789760716.327860:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22028288 chunk: [22028288, 22032384) 22032384, 4096 00020000:00200000:2.0:1789760716.327864:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.327884:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5378, 5378] 00000008:00010000:2.0:1789760716.327902:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.327908:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.327919:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22028288, 22032384) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.327932:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22028288=0x1502000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.328187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.328190:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22032384 00000020:00010000:2.0:1789760716.328200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.328204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.328208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.328211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.328214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.328223:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.328231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.328249:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760716.328369:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22028288<>22032384 00000100:00100000:3.0:1789760716.328779:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.333723:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668992:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.333861:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760716.333869:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493668864:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.333906:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493669120:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.336713:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493668864:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.336739:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493669120:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.336758:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668992:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760716.376373:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972697 gen 3 00000008:00010000:3.0:1789760716.376483:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760716.376492:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.376503:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.376531:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760716.376561:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493668736:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.376654:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5378, 5378] 00000080:00200000:2.0:1789760716.376672:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.376678:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.376695:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22032384 i_blocks 45063 00000020:00200000:2.0:1789760716.376699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.376709:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.376743:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22032384, count: 4096 00000080:00200000:2.0:1789760716.376750:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22032384, bytes: 4096 00000080:00200000:2.0:1789760716.376760:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.376769:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.376772:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22032384, 22036480) 0 0 00000080:00200000:2.0:1789760716.376776:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.376793:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5379, 5379] 00020000:00200000:2.0:1789760716.376801:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22032384 chunk: [22032384, 22036480) 22036480, 4096 00020000:00200000:2.0:1789760716.376805:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.376970:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5379, 5379] 00000008:00010000:2.0:1789760716.376990:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.377004:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.377017:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22032384, 22036480) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.377032:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22032384=0x1503000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.377087:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.377090:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22036480 00000020:00010000:2.0:1789760716.377099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.377103:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.377106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.377109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.377112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.377120:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.377127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.377144:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760716.377244:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22032384<>22036480 00000100:00100000:3.0:1789760716.377447:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.377467:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.407314:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972698 gen 3 00000008:00010000:3.0:1789760716.407367:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760716.407375:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.407388:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.407446:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760716.407476:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669248:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.407624:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5379, 5379] 00000080:00200000:2.0:1789760716.407639:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.407646:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.407661:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22036480 i_blocks 45063 00000020:00200000:2.0:1789760716.407666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.407676:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.407711:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22036480, count: 4096 00000080:00200000:2.0:1789760716.407718:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22036480, bytes: 4096 00000080:00200000:2.0:1789760716.407726:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.407736:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.407738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22036480, 22040576) 0 0 00000080:00200000:2.0:1789760716.407743:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.407758:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5380, 5380] 00020000:00200000:2.0:1789760716.407767:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22036480 chunk: [22036480, 22040576) 22040576, 4096 00020000:00200000:2.0:1789760716.407771:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.407790:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5380, 5380] 00000008:00010000:2.0:1789760716.407810:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.407881:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.407896:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22036480, 22040576) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.407910:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22036480=0x1504000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.408086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.408090:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22040576 00000020:00010000:2.0:1789760716.408098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.408102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.408105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.408109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.408112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.408119:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.408126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.408141:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760716.408238:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 22036480<>22040576 00000100:00100000:3.0:1789760716.408413:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760716.408434:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493669376:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760716.455898:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760716.455918:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.455934:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.455963:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760716.456111:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493669376:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.456253:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5380, 5380] 00000080:00200000:2.0:1789760716.456270:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.456277:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.456293:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22040576 i_blocks 45063 00000020:00200000:2.0:1789760716.456298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.456309:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.456345:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22040576, count: 4096 00000080:00200000:2.0:1789760716.456353:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22040576, bytes: 4096 00000080:00200000:2.0:1789760716.456363:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.456373:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.456375:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22040576, 22044672) 0 0 00000080:00200000:2.0:1789760716.456379:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.456426:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5381, 5381] 00020000:00200000:2.0:1789760716.456435:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22040576 chunk: [22040576, 22044672) 22044672, 4096 00020000:00200000:2.0:1789760716.456439:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.456459:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5381, 5381] 00000008:00010000:2.0:1789760716.456479:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.456486:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.456498:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22040576, 22044672) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.456512:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22040576=0x1505000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.456579:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.456582:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22044672 00000020:00010000:2.0:1789760716.456591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.456595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.456599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.456603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.456606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.456613:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.456620:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.456635:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760716.456735:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 22040576<>22044672 00000100:00100000:3.0:1789760716.457120:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.457142:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.527195:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972700 gen 3 00000008:00010000:3.0:1789760716.527265:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760716.527274:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.527284:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.527306:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760716.527335:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669504:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.527436:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5381, 5381] 00000080:00200000:2.0:1789760716.527453:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.527459:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.527475:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22044672 i_blocks 45063 00000020:00200000:2.0:1789760716.527479:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.527490:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.527535:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22044672, count: 4096 00000080:00200000:2.0:1789760716.527542:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22044672, bytes: 4096 00000080:00200000:2.0:1789760716.527617:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.527627:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.527630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22044672, 22048768) 0 0 00000080:00200000:2.0:1789760716.527635:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.527650:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5382, 5382] 00020000:00200000:2.0:1789760716.527659:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22044672 chunk: [22044672, 22048768) 22048768, 4096 00020000:00200000:2.0:1789760716.527663:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.527686:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5382, 5382] 00000008:00010000:2.0:1789760716.527703:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.527708:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.527720:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22044672, 22048768) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.527735:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22044672=0x1506000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.527797:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.527800:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22048768 00000020:00010000:2.0:1789760716.527808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.527812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.527815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.527819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.527822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.527830:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.527837:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.527854:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760716.527965:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 22044672<>22048768 00000100:00100000:3.0:1789760716.528044:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669632:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.564434:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972701 gen 3 00000008:00010000:3.0:1789760716.564483:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760716.564492:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.564503:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.564531:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760716.564563:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669632:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.564701:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5382, 5382] 00000080:00200000:2.0:1789760716.564719:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.564725:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.564741:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22048768 i_blocks 45063 00000020:00200000:2.0:1789760716.564745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.564757:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.564789:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22048768, count: 4096 00000080:00200000:2.0:1789760716.564798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22048768, bytes: 4096 00000080:00200000:2.0:1789760716.564808:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.564828:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.564830:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22048768, 22052864) 0 0 00000080:00200000:2.0:1789760716.564835:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.564853:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5383, 5383] 00020000:00200000:2.0:1789760716.564862:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22048768 chunk: [22048768, 22052864) 22052864, 4096 00020000:00200000:2.0:1789760716.564866:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.564887:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5383, 5383] 00000008:00010000:2.0:1789760716.564906:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.564913:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.564924:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22048768, 22052864) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.564938:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22048768=0x1507000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.564989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.564991:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22052864 00000020:00010000:2.0:1789760716.565112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.565118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.565121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.565125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.565129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.565138:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.565145:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760716.565222:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:3.0:1789760716.565324:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 22048768<>22052864 00000100:00100000:3.0:1789760716.565363:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.565379:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669760:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760716.605032:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760716.605048:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.605061:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.605083:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760716.605114:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493669760:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.605256:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5383, 5383] 00000080:00200000:2.0:1789760716.605271:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.605278:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.605293:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22052864 i_blocks 45063 00000020:00200000:2.0:1789760716.605297:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.605307:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.605341:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22052864, count: 4096 00000080:00200000:2.0:1789760716.605348:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22052864, bytes: 4096 00000080:00200000:2.0:1789760716.605357:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.605366:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.605368:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22052864, 22056960) 0 0 00000080:00200000:2.0:1789760716.605373:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.605388:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5384, 5384] 00020000:00200000:2.0:1789760716.605426:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22052864 chunk: [22052864, 22056960) 22056960, 4096 00020000:00200000:2.0:1789760716.605431:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.605452:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5384, 5384] 00000008:00010000:2.0:1789760716.605469:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.605475:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.605487:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22052864, 22056960) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.605502:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22052864=0x1508000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.605570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.605573:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22056960 00000020:00010000:2.0:1789760716.605582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.605585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.605589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.605595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.605598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.605606:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.605613:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.605629:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760716.605731:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 22052864<>22056960 00000100:00100000:2.0:1789760716.605774:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760716.605793:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493669888:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.641247:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972703 gen 3 00000008:00010000:3.0:1789760716.641312:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760716.641321:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.641332:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.641354:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760716.641383:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493669888:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.641671:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5384, 5384] 00000080:00200000:2.0:1789760716.641688:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.641694:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.641710:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22056960 i_blocks 45063 00000020:00200000:2.0:1789760716.641714:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.641724:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.641761:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22056960, count: 4096 00000080:00200000:2.0:1789760716.641769:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22056960, bytes: 4096 00000080:00200000:2.0:1789760716.641779:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.641787:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.641790:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22056960, 22061056) 0 0 00000080:00200000:2.0:1789760716.641794:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.641811:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5385, 5385] 00020000:00200000:2.0:1789760716.641829:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22056960 chunk: [22056960, 22061056) 22061056, 4096 00020000:00200000:2.0:1789760716.641835:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.641855:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5385, 5385] 00000008:00010000:2.0:1789760716.641874:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.641880:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.641893:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22056960, 22061056) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.641907:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22056960=0x1509000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.641964:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.641966:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22061056 00000020:00010000:2.0:1789760716.641975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.641979:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.641983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.641986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.641989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.642102:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.642110:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.642125:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760716.642223:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 22056960<>22061056 00000100:00100000:3.0:1789760716.642321:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.642340:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493670016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.681134:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972704 gen 3 00000008:00010000:3.0:1789760716.681188:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760716.681198:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.681210:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.681233:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760716.681267:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493670016:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.681374:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5385, 5385] 00000080:00200000:2.0:1789760716.681390:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.681429:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.681445:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22061056 i_blocks 45063 00000020:00200000:2.0:1789760716.681449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.681462:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.681494:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22061056, count: 4096 00000080:00200000:2.0:1789760716.681501:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22061056, bytes: 4096 00000080:00200000:2.0:1789760716.681511:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.681531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.681534:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22061056, 22065152) 0 0 00000080:00200000:2.0:1789760716.681538:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.681553:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5386, 5386] 00020000:00200000:2.0:1789760716.681562:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22061056 chunk: [22061056, 22065152) 22065152, 4096 00020000:00200000:2.0:1789760716.681566:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.681585:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5386, 5386] 00000008:00010000:2.0:1789760716.681599:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.681615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.681627:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22061056, 22065152) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.681642:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22061056=0x150a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.681697:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.681700:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22065152 00000020:00010000:2.0:1789760716.681708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.681712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.681715:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.681719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.681722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.681731:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.681738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.681756:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760716.681859:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 22061056<>22065152 00000100:00100000:3.0:1789760716.682026:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760716.682046:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493670144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.761265:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972705 gen 3 00000008:00010000:3.0:1789760716.761317:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760716.761326:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.761338:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.761355:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760716.761384:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493670144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760716.761796:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5386, 5386] 00000080:00200000:2.0:1789760716.761813:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760716.761830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760716.761847:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22065152 i_blocks 45063 00000020:00200000:2.0:1789760716.761852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760716.761864:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760716.761899:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22065152, count: 4096 00000080:00200000:2.0:1789760716.761906:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22065152, bytes: 4096 00000080:00200000:2.0:1789760716.761917:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760716.761925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760716.761928:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22065152, 22069248) 0 0 00000080:00200000:2.0:1789760716.761933:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760716.761950:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5387, 5387] 00020000:00200000:2.0:1789760716.761959:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22065152 chunk: [22065152, 22069248) 22069248, 4096 00020000:00200000:2.0:1789760716.761964:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760716.761984:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5387, 5387] 00000008:00010000:2.0:1789760716.762075:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760716.762083:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760716.762095:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22065152, 22069248) append 0 append lockpos 0 00000080:00200000:2.0:1789760716.762109:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22065152=0x150b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760716.762173:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760716.762176:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22069248 00000020:00010000:2.0:1789760716.762186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760716.762191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.762194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760716.762198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760716.762201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760716.762209:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760716.762217:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760716.762244:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760716.762359:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 22065152<>22069248 00000100:00100000:3.0:1789760716.762623:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493670272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760716.815940:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972706 gen 3 00000008:00010000:3.0:1789760716.815992:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760716.816066:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.816079:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.816144:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760716.816174:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493670272:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760717.854983:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5407, 5407] 00000080:00200000:3.0:1789760717.855011:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.855019:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.855034:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22151168 i_blocks 45063 00000020:00200000:3.0:1789760717.855038:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.855049:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.855087:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22151168, count: 4096 00000080:00200000:3.0:1789760717.855094:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22151168, bytes: 4096 00000080:00200000:3.0:1789760717.855104:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.855115:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.855118:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22151168, 22155264) 0 0 00000080:00200000:3.0:1789760717.855123:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.855141:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5408, 5408] 00020000:00200000:3.0:1789760717.855151:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22151168 chunk: [22151168, 22155264) 22155264, 4096 00020000:00200000:3.0:1789760717.855156:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.855179:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5408, 5408] 00000008:00010000:3.0:1789760717.855199:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.855206:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760717.855219:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22151168, 22155264) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.855236:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22151168=0x1520000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.855301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.855304:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22155264 00000020:00010000:3.0:1789760717.855313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.855317:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.855320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.855323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.855327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.855336:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.855342:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.855360:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760717.855549:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 22151168<>22155264 00000100:00100000:3.0:1789760717.855596:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760717.855616:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493672960:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760717.919140:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972727 gen 3 00000008:00010000:3.0:1789760717.919194:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760717.919203:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.919214:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.919295:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760717.919329:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493672960:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760718.060724:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760718.129654:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5412, 5412] 00000080:00200000:3.0:1789760718.129671:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.129677:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.129694:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22171648 i_blocks 45063 00000020:00200000:3.0:1789760718.129698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.129710:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.129754:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22171648, count: 4096 00000080:00200000:3.0:1789760718.129762:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22171648, bytes: 4096 00000080:00200000:3.0:1789760718.129772:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.129781:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.129783:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22171648, 22175744) 0 0 00000080:00200000:3.0:1789760718.129788:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.129804:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5413, 5413] 00020000:00200000:3.0:1789760718.129813:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22171648 chunk: [22171648, 22175744) 22175744, 4096 00020000:00200000:3.0:1789760718.129827:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.129849:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5413, 5413] 00000008:00010000:3.0:1789760718.129869:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.129875:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760718.129887:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22171648, 22175744) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.129902:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22171648=0x1525000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.129972:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.129975:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22175744 00000020:00010000:3.0:1789760718.129993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.129997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.130001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.130004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.130007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.130016:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.130023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.130041:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760718.130141:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 22171648<>22175744 00000100:00100000:2.0:1789760718.130434:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760718.130454:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673600:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760718.173744:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972732 gen 3 00000008:00010000:2.0:1789760718.173826:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760718.173835:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760718.173846:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760718.173865:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760718.173896:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673600:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760718.174008:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5413, 5413] 00000080:00200000:3.0:1789760718.174024:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.174030:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.174046:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22175744 i_blocks 45063 00000020:00200000:3.0:1789760718.174049:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.174058:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.174090:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22175744, count: 4096 00000080:00200000:3.0:1789760718.174098:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22175744, bytes: 4096 00000080:00200000:3.0:1789760718.174107:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.174117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.174119:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22175744, 22179840) 0 0 00000080:00200000:3.0:1789760718.174124:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.174139:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5414, 5414] 00020000:00200000:3.0:1789760718.174147:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22175744 chunk: [22175744, 22179840) 22179840, 4096 00020000:00200000:3.0:1789760718.174151:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.174170:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5414, 5414] 00000008:00010000:3.0:1789760718.174188:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.174195:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760718.174207:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22175744, 22179840) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.174220:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22175744=0x1526000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.174279:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.174283:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22179840 00000020:00010000:3.0:1789760718.174291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.174296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.174299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.174303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.174305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.174313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.174320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.174336:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760718.174475:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 22175744<>22179840 00000100:00100000:2.0:1789760718.174891:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673728:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760718.215199:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972733 gen 3 00000008:00010000:2.0:1789760718.215250:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760718.215259:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760718.215270:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760718.215291:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760718.215323:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673728:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760718.215327:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5414, 5414] 00000080:00200000:3.0:1789760718.215343:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.215349:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.215365:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22179840 i_blocks 45063 00000020:00200000:3.0:1789760718.215369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.215388:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.215447:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22179840, count: 4096 00000080:00200000:3.0:1789760718.215455:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22179840, bytes: 4096 00000080:00200000:3.0:1789760718.215464:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.215472:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.215475:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22179840, 22183936) 0 0 00000080:00200000:3.0:1789760718.215479:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.215515:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5415, 5415] 00020000:00200000:3.0:1789760718.215524:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22179840 chunk: [22179840, 22183936) 22183936, 4096 00020000:00200000:3.0:1789760718.215528:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.215548:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5415, 5415] 00000008:00010000:3.0:1789760718.215576:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.215582:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760718.215594:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22179840, 22183936) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.215607:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22179840=0x1527000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.215659:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.215662:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22183936 00000020:00010000:3.0:1789760718.215669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.215673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.215676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.215679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.215682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.215689:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.215696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.215710:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760718.215808:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22179840<>22183936 00000100:00100000:2.0:1789760718.216020:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760718.216037:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673856:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760718.253367:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972734 gen 3 00000008:00010000:2.0:1789760718.253443:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760718.253453:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760718.253463:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760718.253640:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760718.253671:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673856:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760718.253756:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5415, 5415] 00000080:00200000:3.0:1789760718.253773:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.253779:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.253796:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22183936 i_blocks 45063 00000020:00200000:3.0:1789760718.253800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.253810:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.253910:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22183936, count: 4096 00000080:00200000:3.0:1789760718.253919:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22183936, bytes: 4096 00000080:00200000:3.0:1789760718.253932:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.253940:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.253943:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22183936, 22188032) 0 0 00000080:00200000:3.0:1789760718.253947:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.253964:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5416, 5416] 00020000:00200000:3.0:1789760718.253973:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22183936 chunk: [22183936, 22188032) 22188032, 4096 00020000:00200000:3.0:1789760718.253977:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.254335:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5416, 5416] 00000008:00010000:3.0:1789760718.254355:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.254361:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760718.254372:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22183936, 22188032) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.254387:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22183936=0x1528000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.254479:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.254482:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22188032 00000020:00010000:3.0:1789760718.254491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.254495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.254498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.254621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.254625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.254634:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.254641:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.254663:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760718.254886:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 22183936<>22188032 00000100:00100000:2.0:1789760718.254962:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673984:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760718.322489:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972735 gen 3 00000008:00010000:2.0:1789760718.322555:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760718.322563:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760718.322575:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760718.322599:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760718.322629:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493673984:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760718.970283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5428, 5428] 00000080:00200000:2.0:1789760718.970301:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760718.970306:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760718.970322:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22237184 i_blocks 45063 00000020:00200000:2.0:1789760718.970326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760718.970337:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760718.970372:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22237184, count: 4096 00000080:00200000:2.0:1789760718.970383:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22237184, bytes: 4096 00000080:00200000:2.0:1789760718.970417:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760718.970426:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760718.970428:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22237184, 22241280) 0 0 00000080:00200000:2.0:1789760718.970433:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760718.970450:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5429, 5429] 00020000:00200000:2.0:1789760718.970458:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22237184 chunk: [22237184, 22241280) 22241280, 4096 00020000:00200000:2.0:1789760718.970462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760718.970480:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5429, 5429] 00000008:00010000:2.0:1789760718.970508:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760718.970515:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760718.970527:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22237184, 22241280) append 0 append lockpos 0 00000080:00200000:2.0:1789760718.970541:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22237184=0x1535000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760718.970594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760718.970597:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22241280 00000020:00010000:2.0:1789760718.970604:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760718.970608:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.970611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.970615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.970618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.970625:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.970632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.970646:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760718.970753:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 22237184<>22241280 00000100:00100000:2.0:1789760718.970807:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760718.970949:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493675648:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760719.021454:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760719.021470:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.021483:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.021512:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760719.021542:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493675648:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760719.204482:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5432, 5432] 00000080:00200000:2.0:1789760719.204512:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.204519:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.204535:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22253568 i_blocks 45063 00000020:00200000:2.0:1789760719.204539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.204548:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.204591:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22253568, count: 4096 00000080:00200000:2.0:1789760719.204600:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22253568, bytes: 4096 00000080:00200000:2.0:1789760719.204613:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.204622:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.204624:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22253568, 22257664) 0 0 00000080:00200000:2.0:1789760719.204629:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.204646:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5433, 5433] 00020000:00200000:2.0:1789760719.204655:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22253568 chunk: [22253568, 22257664) 22257664, 4096 00020000:00200000:2.0:1789760719.204660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.204681:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5433, 5433] 00000008:00010000:2.0:1789760719.204701:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.204707:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760719.204729:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22253568, 22257664) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.204744:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22253568=0x1539000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.204795:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.204798:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22257664 00000020:00010000:2.0:1789760719.204807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.204810:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.204812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.204816:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.204819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.204826:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.204832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.272269:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5433, 5433] 00000080:00200000:2.0:1789760719.272286:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.272301:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.272318:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22257664 i_blocks 45063 00000020:00200000:2.0:1789760719.272322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.272333:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.272371:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22257664, count: 4096 00000080:00200000:2.0:1789760719.272379:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22257664, bytes: 4096 00000080:00200000:2.0:1789760719.272391:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.272426:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.272430:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22257664, 22261760) 0 0 00000080:00200000:2.0:1789760719.272434:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.272452:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5434, 5434] 00020000:00200000:2.0:1789760719.272461:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22257664 chunk: [22257664, 22261760) 22261760, 4096 00020000:00200000:2.0:1789760719.272465:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.272487:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5434, 5434] 00000008:00010000:2.0:1789760719.272515:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.272522:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760719.272535:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22257664, 22261760) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.272549:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22257664=0x153a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.272604:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.272607:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22261760 00000020:00010000:2.0:1789760719.272616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.272620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.272623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.272626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.272630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.272639:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.272646:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.272663:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760719.272784:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22257664<>22261760 00000100:00100000:2.0:1789760719.272839:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760719.272859:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493676288:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760719.336968:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972753 gen 3 00000008:00010000:2.0:1789760719.337033:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760719.337042:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.337055:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.337079:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760719.337110:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493676288:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760719.821432:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5441, 5441] 00000080:00200000:3.0:1789760719.821447:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.821452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.821468:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22290432 i_blocks 45063 00000020:00200000:3.0:1789760719.821471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.821480:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.821522:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22290432, count: 4096 00000080:00200000:3.0:1789760719.821529:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22290432, bytes: 4096 00000080:00200000:3.0:1789760719.821544:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.821554:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.821557:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22290432, 22294528) 0 0 00000080:00200000:3.0:1789760719.821561:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.821577:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5442, 5442] 00020000:00200000:3.0:1789760719.821585:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22290432 chunk: [22290432, 22294528) 22294528, 4096 00020000:00200000:3.0:1789760719.821589:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.821608:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5442, 5442] 00000008:00010000:3.0:1789760719.821626:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.821632:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760719.821643:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22290432, 22294528) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.821657:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22290432=0x1542000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.821710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.821713:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22294528 00000020:00010000:3.0:1789760719.821722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.821726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.821729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.821732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.821735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.821742:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.821749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.821765:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760719.821868:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 22290432<>22294528 00000100:00100000:3.0:1789760719.821915:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760719.821933:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493677312:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760719.902346:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972761 gen 3 00000008:00010000:2.0:1789760719.902444:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760719.902453:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.902464:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.902507:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760719.902536:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493677312:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760719.902985:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5442, 5442] 00000080:00200000:3.0:1789760719.903003:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.903008:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.903022:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22294528 i_blocks 45063 00000020:00200000:3.0:1789760719.903026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.903035:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.903067:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22294528, count: 4096 00000080:00200000:3.0:1789760719.903074:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22294528, bytes: 4096 00000080:00200000:3.0:1789760719.903084:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.903093:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.903095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22294528, 22298624) 0 0 00000080:00200000:3.0:1789760719.903100:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.903115:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5443, 5443] 00020000:00200000:3.0:1789760719.903124:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22294528 chunk: [22294528, 22298624) 22298624, 4096 00020000:00200000:3.0:1789760719.903128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.903147:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5443, 5443] 00000008:00010000:3.0:1789760719.903165:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.903171:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760719.903183:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22294528, 22298624) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.903196:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22294528=0x1543000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.903252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.903255:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22298624 00000020:00010000:3.0:1789760719.903263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.903266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.903268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.903272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.903274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.903281:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.903286:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.903301:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760719.903439:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 22294528<>22298624 00000100:00100000:2.0:1789760719.903729:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760719.903752:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677440:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760719.969348:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972762 gen 3 00000008:00010000:2.0:1789760719.969440:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760719.969450:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.969462:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.969538:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760719.969567:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677440:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760719.971773:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5443, 5443] 00000080:00200000:3.0:1789760719.971799:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.971806:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.971905:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22298624 i_blocks 45063 00000020:00200000:3.0:1789760719.971911:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.971927:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.971961:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22298624, count: 4096 00000080:00200000:3.0:1789760719.972032:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22298624, bytes: 4096 00000080:00200000:3.0:1789760719.972049:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.972058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.972060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22298624, 22302720) 0 0 00000080:00200000:3.0:1789760719.972065:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.972081:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5444, 5444] 00020000:00200000:3.0:1789760719.972089:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22298624 chunk: [22298624, 22302720) 22302720, 4096 00020000:00200000:3.0:1789760719.972094:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.972228:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5444, 5444] 00000008:00010000:3.0:1789760719.972248:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.972254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760719.972266:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22298624, 22302720) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.972281:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22298624=0x1544000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.972345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.972348:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22302720 00000020:00010000:3.0:1789760719.972356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.972360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.972363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.972367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.972369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.972379:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.972385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.972431:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760719.972631:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 22298624<>22302720 00000100:00100000:2.0:1789760719.972722:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677568:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.051031:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972763 gen 3 00000008:00010000:2.0:1789760720.051086:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760720.051096:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.051115:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.051230:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760720.051259:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677568:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.051731:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5444, 5444] 00000080:00200000:3.0:1789760720.051749:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.051755:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.051785:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22302720 i_blocks 45063 00000020:00200000:3.0:1789760720.051789:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.051798:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.051838:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22302720, count: 4096 00000080:00200000:3.0:1789760720.051845:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22302720, bytes: 4096 00000080:00200000:3.0:1789760720.051859:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.051867:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.051870:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22302720, 22306816) 0 0 00000080:00200000:3.0:1789760720.051875:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.051890:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5445, 5445] 00020000:00200000:3.0:1789760720.051898:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22302720 chunk: [22302720, 22306816) 22306816, 4096 00020000:00200000:3.0:1789760720.051903:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.051920:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5445, 5445] 00000008:00010000:3.0:1789760720.051938:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.051945:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.051957:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22302720, 22306816) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.052172:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22302720=0x1545000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.052228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.052230:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22306816 00000020:00010000:3.0:1789760720.052238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.052241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.052244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.052248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.052251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.052258:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.052264:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.052278:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760720.052369:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 22302720<>22306816 00000100:00100000:2.0:1789760720.052487:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493677696:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.152361:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972764 gen 3 00000008:00010000:2.0:1789760720.152436:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760720.152445:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.152457:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.152476:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760720.152514:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493677696:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.152607:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5445, 5445] 00000080:00200000:3.0:1789760720.152623:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.152628:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.152644:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22306816 i_blocks 45063 00000020:00200000:3.0:1789760720.152648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.152658:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.152692:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22306816, count: 4096 00000080:00200000:3.0:1789760720.152699:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22306816, bytes: 4096 00000080:00200000:3.0:1789760720.152707:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.152716:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.152719:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22306816, 22310912) 0 0 00000080:00200000:3.0:1789760720.152723:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.152739:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5446, 5446] 00020000:00200000:3.0:1789760720.152748:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22306816 chunk: [22306816, 22310912) 22310912, 4096 00020000:00200000:3.0:1789760720.152752:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.152768:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5446, 5446] 00000008:00010000:3.0:1789760720.152785:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.152790:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.152801:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22306816, 22310912) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.152814:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22306816=0x1546000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.152865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.152867:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22310912 00000020:00010000:3.0:1789760720.152875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.152878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.152881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.152885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.152888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.152895:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.152901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.152914:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760720.153016:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 22306816<>22310912 00000100:00100000:2.0:1789760720.155625:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677824:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.194367:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972765 gen 3 00000008:00010000:2.0:1789760720.194447:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760720.194456:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.194466:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.194485:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760720.194591:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677824:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.194685:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5446, 5446] 00000080:00200000:3.0:1789760720.194702:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.194708:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.194721:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22310912 i_blocks 45063 00000020:00200000:3.0:1789760720.194725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.194734:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.194766:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22310912, count: 4096 00000080:00200000:3.0:1789760720.194773:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22310912, bytes: 4096 00000080:00200000:3.0:1789760720.194781:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.194789:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.194791:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22310912, 22315008) 0 0 00000080:00200000:3.0:1789760720.194796:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.194812:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5447, 5447] 00020000:00200000:3.0:1789760720.194820:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22310912 chunk: [22310912, 22315008) 22315008, 4096 00020000:00200000:3.0:1789760720.194825:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.194843:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5447, 5447] 00000008:00010000:3.0:1789760720.194861:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.194867:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.194878:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22310912, 22315008) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.194890:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22310912=0x1547000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.194944:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.194947:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22315008 00000020:00010000:3.0:1789760720.194955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.194958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.194961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.194965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.194967:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.194973:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.194979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.194994:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760720.195091:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 22310912<>22315008 00000100:00100000:2.0:1789760720.195155:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760720.195183:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677952:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.268949:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972766 gen 3 00000008:00010000:2.0:1789760720.269031:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760720.269043:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.269059:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.269081:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760720.269115:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493677952:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.269345:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5447, 5447] 00000080:00200000:3.0:1789760720.269366:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.269373:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.269391:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22315008 i_blocks 45063 00000020:00200000:3.0:1789760720.269423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.269434:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.269472:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22315008, count: 4096 00000080:00200000:3.0:1789760720.269480:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22315008, bytes: 4096 00000080:00200000:3.0:1789760720.269501:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.269510:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.269513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22315008, 22319104) 0 0 00000080:00200000:3.0:1789760720.269518:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.269535:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5448, 5448] 00020000:00200000:3.0:1789760720.269610:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22315008 chunk: [22315008, 22319104) 22319104, 4096 00020000:00200000:3.0:1789760720.269616:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.269638:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5448, 5448] 00000008:00010000:3.0:1789760720.269656:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.269662:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.269674:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22315008, 22319104) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.269687:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22315008=0x1548000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.269749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.269752:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22319104 00000020:00010000:3.0:1789760720.269761:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.269765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.269768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.269771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.269774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.269782:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.269788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.269805:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760720.269911:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 22315008<>22319104 00000100:00100000:2.0:1789760720.271454:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760720.271503:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678080:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.349412:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972767 gen 3 00000008:00010000:2.0:1789760720.349475:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760720.349485:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.349505:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.349527:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760720.349563:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678080:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.349816:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5448, 5448] 00000080:00200000:3.0:1789760720.349835:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.349842:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.349861:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22319104 i_blocks 45063 00000020:00200000:3.0:1789760720.349865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.349875:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.349911:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22319104, count: 4096 00000080:00200000:3.0:1789760720.349919:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22319104, bytes: 4096 00000080:00200000:3.0:1789760720.349930:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.349939:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.349941:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22319104, 22323200) 0 0 00000080:00200000:3.0:1789760720.349946:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.349963:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5449, 5449] 00020000:00200000:3.0:1789760720.349982:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22319104 chunk: [22319104, 22323200) 22323200, 4096 00020000:00200000:3.0:1789760720.349987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.350011:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5449, 5449] 00000008:00010000:3.0:1789760720.350031:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.350039:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.350052:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22319104, 22323200) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.350068:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22319104=0x1549000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.350130:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.350133:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22323200 00000020:00010000:3.0:1789760720.350141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.350145:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.350149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.350153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.350156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.350164:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.350172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.350188:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760720.350289:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22319104<>22323200 00000100:00100000:2.0:1789760720.350828:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760720.350853:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678208:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.403138:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972768 gen 3 00000008:00010000:2.0:1789760720.403188:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760720.403197:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.403209:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.403230:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760720.403260:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678208:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.403433:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5449, 5449] 00000080:00200000:3.0:1789760720.403449:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.403455:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.403472:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22323200 i_blocks 45063 00000020:00200000:3.0:1789760720.403476:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.403485:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.403527:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22323200, count: 4096 00000080:00200000:3.0:1789760720.403535:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22323200, bytes: 4096 00000080:00200000:3.0:1789760720.403553:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.403563:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.403566:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22323200, 22327296) 0 0 00000080:00200000:3.0:1789760720.403570:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.403586:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5450, 5450] 00020000:00200000:3.0:1789760720.403594:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22323200 chunk: [22323200, 22327296) 22327296, 4096 00020000:00200000:3.0:1789760720.403599:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.403617:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5450, 5450] 00000008:00010000:3.0:1789760720.403634:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.403640:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.403651:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22323200, 22327296) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.403673:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22323200=0x154a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.403870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.403873:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22327296 00000020:00010000:3.0:1789760720.403881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.403885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.403888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.403892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.403894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.403902:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.403908:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.403923:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760720.404029:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 22323200<>22327296 00000100:00100000:2.0:1789760720.404170:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760720.404189:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678336:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.447003:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972769 gen 3 00000008:00010000:2.0:1789760720.447058:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760720.447067:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.447078:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.447096:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760720.447126:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678336:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.447343:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5450, 5450] 00000080:00200000:3.0:1789760720.447360:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.447367:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.447382:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22327296 i_blocks 45063 00000020:00200000:3.0:1789760720.447385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.447415:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.447446:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22327296, count: 4096 00000080:00200000:3.0:1789760720.447462:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22327296, bytes: 4096 00000080:00200000:3.0:1789760720.447471:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.447480:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.447482:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22327296, 22331392) 0 0 00000080:00200000:3.0:1789760720.447495:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.447510:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5451, 5451] 00020000:00200000:3.0:1789760720.447518:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22327296 chunk: [22327296, 22331392) 22331392, 4096 00020000:00200000:3.0:1789760720.447523:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.447540:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5451, 5451] 00000008:00010000:3.0:1789760720.447558:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.447565:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.447576:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22327296, 22331392) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.447590:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22327296=0x154b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.447646:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.447649:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22331392 00000020:00010000:3.0:1789760720.447668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.447672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.447675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.447679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.447682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.447690:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.447697:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.447894:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760720.448005:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 22327296<>22331392 00000100:00100000:2.0:1789760720.448067:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678464:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.514005:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972770 gen 3 00000008:00010000:2.0:1789760720.514062:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760720.514071:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.514081:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.514104:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760720.514135:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493678464:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.518743:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5451, 5451] 00000080:00200000:3.0:1789760720.518770:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.518777:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.518796:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22331392 i_blocks 45063 00000020:00200000:3.0:1789760720.518802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.519077:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.519112:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22331392, count: 4096 00000080:00200000:3.0:1789760720.519120:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22331392, bytes: 4096 00000080:00200000:3.0:1789760720.519135:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.519145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.519147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22331392, 22335488) 0 0 00000080:00200000:3.0:1789760720.519152:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.519169:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5452, 5452] 00020000:00200000:3.0:1789760720.519178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22331392 chunk: [22331392, 22335488) 22335488, 4096 00020000:00200000:3.0:1789760720.519183:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.519204:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5452, 5452] 00000008:00010000:3.0:1789760720.519225:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.519232:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.520475:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22331392, 22335488) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.520521:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22331392=0x154c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.522048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.522060:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22335488 00000020:00010000:3.0:1789760720.522078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.522083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.522088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.522093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.522096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.522111:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.522122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.522160:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760720.522305:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 22331392<>22335488 00000100:00100000:2.0:1789760720.522790:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678592:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.580480:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972771 gen 3 00000008:00010000:2.0:1789760720.580528:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760720.580534:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.580542:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.580560:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760720.580585:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678592:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.580868:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5452, 5452] 00000080:00200000:3.0:1789760720.580882:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.580886:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.580898:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22335488 i_blocks 45063 00000020:00200000:3.0:1789760720.580902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.580910:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.580938:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22335488, count: 4096 00000080:00200000:3.0:1789760720.580944:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22335488, bytes: 4096 00000080:00200000:3.0:1789760720.580952:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.580959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.580961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22335488, 22339584) 0 0 00000080:00200000:3.0:1789760720.580965:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.580977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5453, 5453] 00020000:00200000:3.0:1789760720.580984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22335488 chunk: [22335488, 22339584) 22339584, 4096 00020000:00200000:3.0:1789760720.580987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.581002:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5453, 5453] 00000008:00010000:3.0:1789760720.581014:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.581019:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.581027:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22335488, 22339584) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.581038:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22335488=0x154d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.582084:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.582089:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22339584 00000020:00010000:3.0:1789760720.582100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.582103:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.582106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.582109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.582111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.582119:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.582125:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.582145:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760720.582224:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 22335488<>22339584 00000100:00100000:2.0:1789760720.582656:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760720.582676:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678720:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.662767:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972772 gen 3 00000008:00010000:2.0:1789760720.662800:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760720.662806:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.662814:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.662831:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760720.662856:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678720:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.663005:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5453, 5453] 00000080:00200000:3.0:1789760720.663019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.663024:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.663036:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22339584 i_blocks 45063 00000020:00200000:3.0:1789760720.663039:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.663047:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.663073:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22339584, count: 4096 00000080:00200000:3.0:1789760720.663079:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22339584, bytes: 4096 00000080:00200000:3.0:1789760720.663088:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.663095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.663097:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22339584, 22343680) 0 0 00000080:00200000:3.0:1789760720.663100:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.663113:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5454, 5454] 00020000:00200000:3.0:1789760720.663120:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22339584 chunk: [22339584, 22343680) 22343680, 4096 00020000:00200000:3.0:1789760720.663123:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.663136:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5454, 5454] 00000008:00010000:3.0:1789760720.663149:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.663154:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.663163:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22339584, 22343680) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.663175:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22339584=0x154e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.663220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.663222:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22343680 00000020:00010000:3.0:1789760720.663228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.663231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.663234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.663237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.663240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.663246:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.663251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.663265:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760720.663342:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 22339584<>22343680 00000100:00100000:2.0:1789760720.663442:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678848:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.740578:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972773 gen 3 00000008:00010000:2.0:1789760720.740613:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760720.740619:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.740627:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.740641:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760720.740662:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678848:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.740838:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5454, 5454] 00000080:00200000:3.0:1789760720.740851:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.740856:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.740868:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22343680 i_blocks 45063 00000020:00200000:3.0:1789760720.740871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.740878:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.740905:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22343680, count: 4096 00000080:00200000:3.0:1789760720.740912:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22343680, bytes: 4096 00000080:00200000:3.0:1789760720.740918:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.740924:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.740926:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22343680, 22347776) 0 0 00000080:00200000:3.0:1789760720.740929:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.740943:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5455, 5455] 00020000:00200000:3.0:1789760720.740950:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22343680 chunk: [22343680, 22347776) 22347776, 4096 00020000:00200000:3.0:1789760720.740954:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.740981:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5455, 5455] 00000008:00010000:3.0:1789760720.740996:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.741001:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.741009:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22343680, 22347776) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.741019:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22343680=0x154f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.741076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.741079:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22347776 00000020:00010000:3.0:1789760720.741086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.741089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.741093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.741097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.741099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.741106:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.741113:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.741129:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760720.741207:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 22343680<>22347776 00000100:00100000:2.0:1789760720.741431:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760720.741447:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678976:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.789199:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972774 gen 3 00000008:00010000:2.0:1789760720.789251:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760720.789258:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.789269:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.789291:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760720.789321:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493678976:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.789483:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5455, 5455] 00000080:00200000:3.0:1789760720.789508:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.789514:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.789530:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22347776 i_blocks 45063 00000020:00200000:3.0:1789760720.789542:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.789553:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.789684:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22347776, count: 4096 00000080:00200000:3.0:1789760720.789694:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22347776, bytes: 4096 00000080:00200000:3.0:1789760720.789706:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.789715:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.789717:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22347776, 22351872) 0 0 00000080:00200000:3.0:1789760720.789722:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.789740:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5456, 5456] 00020000:00200000:3.0:1789760720.789749:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22347776 chunk: [22347776, 22351872) 22351872, 4096 00020000:00200000:3.0:1789760720.789753:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.789773:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5456, 5456] 00000008:00010000:3.0:1789760720.789792:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.789798:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.789810:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22347776, 22351872) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.789930:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22347776=0x1550000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.790055:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.790058:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22351872 00000020:00010000:3.0:1789760720.790066:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.790069:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.790072:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.790076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.790078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.790087:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.790094:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.790111:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760720.790217:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22347776<>22351872 00000100:00100000:3.0:1789760720.790261:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760720.790281:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679104:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760720.930585:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760720.930614:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.930626:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.930650:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760720.930680:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679104:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.930824:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5456, 5456] 00000080:00200000:3.0:1789760720.930852:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.930859:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.930876:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22351872 i_blocks 45063 00000020:00200000:3.0:1789760720.930879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.930890:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.930922:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22351872, count: 4096 00000080:00200000:3.0:1789760720.930929:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22351872, bytes: 4096 00000080:00200000:3.0:1789760720.930939:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.930947:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.930950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22351872, 22355968) 0 0 00000080:00200000:3.0:1789760720.930954:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.930981:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5457, 5457] 00020000:00200000:3.0:1789760720.930989:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22351872 chunk: [22351872, 22355968) 22355968, 4096 00020000:00200000:3.0:1789760720.930994:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.931015:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5457, 5457] 00000008:00010000:3.0:1789760720.931033:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.931040:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.931052:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22351872, 22355968) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.931067:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22351872=0x1551000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.931309:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.931312:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22355968 00000020:00010000:3.0:1789760720.931322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.931326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.931330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.931334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.931337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.931345:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.931352:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.931369:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760720.931542:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 22351872<>22355968 00000100:00100000:2.0:1789760720.932221:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760720.932246:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679232:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760720.998602:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972775 gen 3 00000100:00100000:2.0:1789760720.998647:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149880 x1876699493679232/t4294972776(4294972776) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760736 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760720.998708:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760720.998716:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760720.998727:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760720.998749:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760720.998776:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679232:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760720.999139:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5457, 5457] 00000080:00200000:3.0:1789760720.999155:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760720.999161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760720.999175:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22355968 i_blocks 45063 00000020:00200000:3.0:1789760720.999179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760720.999188:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760720.999220:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22355968, count: 4096 00000080:00200000:3.0:1789760720.999227:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22355968, bytes: 4096 00000080:00200000:3.0:1789760720.999236:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760720.999245:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760720.999248:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22355968, 22360064) 0 0 00000080:00200000:3.0:1789760720.999252:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760720.999267:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5458, 5458] 00020000:00200000:3.0:1789760720.999275:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22355968 chunk: [22355968, 22360064) 22360064, 4096 00020000:00200000:3.0:1789760720.999280:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760720.999298:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5458, 5458] 00000008:00010000:3.0:1789760720.999314:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760720.999321:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760720.999333:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22355968, 22360064) append 0 append lockpos 0 00000080:00200000:3.0:1789760720.999346:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22355968=0x1552000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760720.999432:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760720.999435:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22360064 00000020:00010000:3.0:1789760720.999443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760720.999447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.999451:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760720.999455:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760720.999458:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760720.999466:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760720.999473:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760720.999499:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760720.999614:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 22355968<>22360064 00000100:00100000:2.0:1789760720.999752:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760720.999773:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679360:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.041236:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972777 gen 3 00000008:00010000:2.0:1789760721.041301:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760721.041310:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.041321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.041343:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760721.041382:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679360:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.041563:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5458, 5458] 00000080:00200000:3.0:1789760721.041579:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.041586:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.041603:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22360064 i_blocks 45063 00000020:00200000:3.0:1789760721.041608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.041616:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.041650:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22360064, count: 4096 00000080:00200000:3.0:1789760721.041657:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22360064, bytes: 4096 00000080:00200000:3.0:1789760721.041667:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.041676:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.041678:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22360064, 22364160) 0 0 00000080:00200000:3.0:1789760721.041683:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.041702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5459, 5459] 00020000:00200000:3.0:1789760721.041710:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22360064 chunk: [22360064, 22364160) 22364160, 4096 00020000:00200000:3.0:1789760721.041715:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.041735:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5459, 5459] 00000008:00010000:3.0:1789760721.041753:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.041770:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.041783:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22360064, 22364160) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.041799:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22360064=0x1553000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.041857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.041859:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22364160 00000020:00010000:3.0:1789760721.041868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.041871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.041874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.041878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.041880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.041888:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.041895:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.041912:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760721.042009:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 22360064<>22364160 00000100:00100000:2.0:1789760721.042193:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760721.042222:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679488:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.119663:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972778 gen 3 00000008:00010000:2.0:1789760721.119749:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760721.119758:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.119770:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.119869:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760721.119894:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679488:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.124002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5459, 5459] 00000080:00200000:3.0:1789760721.124033:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.124040:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.124061:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22364160 i_blocks 45063 00000020:00200000:3.0:1789760721.124067:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.124085:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.124119:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22364160, count: 4096 00000080:00200000:3.0:1789760721.124129:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22364160, bytes: 4096 00000080:00200000:3.0:1789760721.124144:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.124154:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.124157:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22364160, 22368256) 0 0 00000080:00200000:3.0:1789760721.124162:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.124179:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5460, 5460] 00020000:00200000:3.0:1789760721.124188:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22364160 chunk: [22364160, 22368256) 22368256, 4096 00020000:00200000:3.0:1789760721.124193:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.124213:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5460, 5460] 00000008:00010000:3.0:1789760721.124233:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.124239:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.124251:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22364160, 22368256) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.124266:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22364160=0x1554000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.124350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.124353:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22368256 00000020:00010000:3.0:1789760721.124361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.124365:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.124368:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.124372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.124375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.124384:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.124391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.124451:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760721.124576:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 22364160<>22368256 00000100:00100000:2.0:1789760721.125222:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760721.125251:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679616:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760721.132786:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:2.0:1789760721.169513:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760721.169526:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.169544:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.169560:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760721.169588:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679616:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.169719:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5460, 5460] 00000080:00200000:3.0:1789760721.169734:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.169740:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.169754:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22368256 i_blocks 45063 00000020:00200000:3.0:1789760721.169758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.169767:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.169798:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22368256, count: 4096 00000080:00200000:3.0:1789760721.169805:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22368256, bytes: 4096 00000080:00200000:3.0:1789760721.169814:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.169822:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.169824:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22368256, 22372352) 0 0 00000080:00200000:3.0:1789760721.169829:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.169844:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5461, 5461] 00020000:00200000:3.0:1789760721.169852:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22368256 chunk: [22368256, 22372352) 22372352, 4096 00020000:00200000:3.0:1789760721.169857:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.169875:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5461, 5461] 00000008:00010000:3.0:1789760721.169891:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.169896:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.169907:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22368256, 22372352) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.169931:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22368256=0x1555000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.169989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.169992:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22372352 00000020:00010000:3.0:1789760721.170000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.170004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.170007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.170011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.170014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.170021:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.170026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.170041:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760721.170147:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 22368256<>22372352 00000100:00100000:2.0:1789760721.170449:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760721.170469:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679744:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.225619:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972780 gen 3 00000008:00010000:2.0:1789760721.225713:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760721.225722:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.225733:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.225754:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760721.225785:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493679744:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.225851:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5461, 5461] 00000080:00200000:3.0:1789760721.225869:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.225874:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.225892:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22372352 i_blocks 45063 00000020:00200000:3.0:1789760721.225896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.225906:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.225944:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22372352, count: 4096 00000080:00200000:3.0:1789760721.225951:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22372352, bytes: 4096 00000080:00200000:3.0:1789760721.225960:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.225970:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.225972:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22372352, 22376448) 0 0 00000080:00200000:3.0:1789760721.225977:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.225994:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5462, 5462] 00020000:00200000:3.0:1789760721.226003:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22372352 chunk: [22372352, 22376448) 22376448, 4096 00020000:00200000:3.0:1789760721.226008:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.226030:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5462, 5462] 00000008:00010000:3.0:1789760721.226058:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.226066:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.226077:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22372352, 22376448) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.226091:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22372352=0x1556000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.226146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.226149:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22376448 00000020:00010000:3.0:1789760721.226158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.226162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.226165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.226169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.226172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.226180:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.226187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.226204:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760721.226312:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 22372352<>22376448 00000100:00100000:2.0:1789760721.227001:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760721.227024:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679872:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760721.303228:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760721.303244:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.303257:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.303281:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000080:00200000:3.0:1789760721.304068:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5462, 5462] 00000080:00200000:3.0:1789760721.304089:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.304095:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.304114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22376448 i_blocks 45063 00000020:00200000:3.0:1789760721.304119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.304133:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:2.0:1789760721.304155:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493679872:192.168.203.160@tcp:4:dd.0 00000080:00200008:3.0:1789760721.304171:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22376448, count: 4096 00000080:00200000:3.0:1789760721.304182:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22376448, bytes: 4096 00000080:00200000:3.0:1789760721.304195:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.304205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.304207:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22376448, 22380544) 0 0 00000080:00200000:3.0:1789760721.304213:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.304230:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5463, 5463] 00020000:00200000:3.0:1789760721.304241:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22376448 chunk: [22376448, 22380544) 22380544, 4096 00020000:00200000:3.0:1789760721.304246:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.304267:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5463, 5463] 00000008:00010000:3.0:1789760721.304287:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.304294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.304307:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22376448, 22380544) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.304322:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22376448=0x1557000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.304389:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.304410:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22380544 00000020:00010000:3.0:1789760721.304420:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.304424:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.304427:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.304431:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.304435:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.304442:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.304449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.304465:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760721.304577:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 22376448<>22380544 00000100:00100000:2.0:1789760721.304678:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760721.304695:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680000:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.349098:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972782 gen 3 00000008:00010000:2.0:1789760721.349161:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760721.349170:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.349180:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.349241:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760721.349273:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680000:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.349737:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5463, 5463] 00000080:00200000:3.0:1789760721.349875:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.349883:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.349899:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22380544 i_blocks 45063 00000020:00200000:3.0:1789760721.349903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.349914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.349948:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22380544, count: 4096 00000080:00200000:3.0:1789760721.350002:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22380544, bytes: 4096 00000080:00200000:3.0:1789760721.350016:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.350024:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.350026:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22380544, 22384640) 0 0 00000080:00200000:3.0:1789760721.350031:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.350047:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5464, 5464] 00020000:00200000:3.0:1789760721.350056:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22380544 chunk: [22380544, 22384640) 22384640, 4096 00020000:00200000:3.0:1789760721.350061:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.350083:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5464, 5464] 00000008:00010000:3.0:1789760721.350102:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.350108:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.350119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22380544, 22384640) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.350134:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22380544=0x1558000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.350196:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.350199:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22384640 00000020:00010000:3.0:1789760721.350208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.350212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.350215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.350218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.350221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.350229:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.350237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.350258:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760721.350367:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 22380544<>22384640 00000100:00100000:2.0:1789760721.350449:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680128:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760721.387956:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760721.388068:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.388080:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.388101:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760721.388132:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680128:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.388224:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5464, 5464] 00000080:00200000:3.0:1789760721.388241:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.388247:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.388262:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22384640 i_blocks 45063 00000020:00200000:3.0:1789760721.388267:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.388276:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.388310:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22384640, count: 4096 00000080:00200000:3.0:1789760721.388318:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22384640, bytes: 4096 00000080:00200000:3.0:1789760721.388329:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.388339:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.388341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22384640, 22388736) 0 0 00000080:00200000:3.0:1789760721.388346:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.388362:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5465, 5465] 00020000:00200000:3.0:1789760721.388371:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22384640 chunk: [22384640, 22388736) 22388736, 4096 00020000:00200000:3.0:1789760721.388376:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.388506:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5465, 5465] 00000008:00010000:3.0:1789760721.388525:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.388531:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.388553:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22384640, 22388736) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.388568:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22384640=0x1559000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.388631:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.388634:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22388736 00000020:00010000:3.0:1789760721.388642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.388646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.388649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.388653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.388655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.388663:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.388670:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.388689:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760721.388786:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 22384640<>22388736 00000100:00100000:2.0:1789760721.388847:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760721.389205:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680256:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.419888:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972784 gen 3 00000008:00010000:2.0:1789760721.419953:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760721.419962:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.419983:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.420004:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760721.420035:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680256:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.420238:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5465, 5465] 00000080:00200000:3.0:1789760721.420255:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.420262:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.420279:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22388736 i_blocks 45063 00000020:00200000:3.0:1789760721.420283:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.420293:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.420331:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22388736, count: 4096 00000080:00200000:3.0:1789760721.420339:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22388736, bytes: 4096 00000080:00200000:3.0:1789760721.420351:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.420360:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.420363:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22388736, 22392832) 0 0 00000080:00200000:3.0:1789760721.420367:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.420384:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5466, 5466] 00020000:00200000:3.0:1789760721.420424:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22388736 chunk: [22388736, 22392832) 22392832, 4096 00020000:00200000:3.0:1789760721.420430:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.420452:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5466, 5466] 00000008:00010000:3.0:1789760721.420473:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.420479:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.420503:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22388736, 22392832) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.420517:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22388736=0x155a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.420575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.420578:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22392832 00000020:00010000:3.0:1789760721.420587:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.420590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.420594:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.420597:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.420600:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.420608:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.420615:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.420631:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760721.420735:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 22388736<>22392832 00000100:00100000:3.0:1789760721.420776:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760721.420794:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493680384:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.449791:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972785 gen 3 00000008:00010000:2.0:1789760721.449853:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760721.449863:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760721.449874:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760721.449892:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760721.449921:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493680384:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760721.450275:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5466, 5466] 00000080:00200000:3.0:1789760721.450292:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.450298:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.450315:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22392832 i_blocks 45063 00000020:00200000:3.0:1789760721.450320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.450330:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.450364:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22392832, count: 4096 00000080:00200000:3.0:1789760721.450371:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22392832, bytes: 4096 00000080:00200000:3.0:1789760721.450382:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.450424:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.450427:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22392832, 22396928) 0 0 00000080:00200000:3.0:1789760721.450432:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.450451:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5467, 5467] 00020000:00200000:3.0:1789760721.450460:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22392832 chunk: [22392832, 22396928) 22396928, 4096 00020000:00200000:3.0:1789760721.450465:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.450495:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5467, 5467] 00000008:00010000:3.0:1789760721.450516:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.450522:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760721.450533:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22392832, 22396928) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.450546:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22392832=0x155b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.450604:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.450607:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22396928 00000020:00010000:3.0:1789760721.450615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.450618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.450621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.450625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.450627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.450635:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.450642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.450658:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760721.450762:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22392832<>22396928 00000100:00100000:2.0:1789760721.450897:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493680512:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760721.452784:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493680640:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760721.452896:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493680896:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760721.452936:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760721.452943:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680768:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760721.455573:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493680768:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760721.455594:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493680896:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760721.457879:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493680640:192.168.203.160@tcp:400:kworker.0 00000008:00010000:3.0:1789760721.492768:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760721.492787:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.492799:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.492827:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760721.492858:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493680512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.493107:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5467, 5467] 00000080:00200000:2.0:1789760721.493125:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.493131:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.493147:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22396928 i_blocks 45063 00000020:00200000:2.0:1789760721.493151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.493161:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.493194:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22396928, count: 4096 00000080:00200000:2.0:1789760721.493202:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22396928, bytes: 4096 00000080:00200000:2.0:1789760721.493213:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.493222:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.493224:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22396928, 22401024) 0 0 00000080:00200000:2.0:1789760721.493228:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.493245:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5468, 5468] 00020000:00200000:2.0:1789760721.493256:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22396928 chunk: [22396928, 22401024) 22401024, 4096 00020000:00200000:2.0:1789760721.493260:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.493279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5468, 5468] 00000008:00010000:2.0:1789760721.493297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.493304:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.493316:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22396928, 22401024) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.493332:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22396928=0x155c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.493434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.493437:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22401024 00000020:00010000:2.0:1789760721.493446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.493449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.493452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.493456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.493459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.493467:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.493474:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.493500:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760721.493607:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22396928<>22401024 00000100:00100000:3.0:1789760721.493887:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.545115:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972787 gen 3 00000008:00010000:3.0:1789760721.545181:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760721.545191:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.545202:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.545225:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760721.545254:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.545493:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5468, 5468] 00000080:00200000:2.0:1789760721.545511:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.545517:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.545533:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22401024 i_blocks 45063 00000020:00200000:2.0:1789760721.545537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.545548:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.545584:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22401024, count: 4096 00000080:00200000:2.0:1789760721.545592:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22401024, bytes: 4096 00000080:00200000:2.0:1789760721.545602:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.545611:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.545613:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22401024, 22405120) 0 0 00000080:00200000:2.0:1789760721.545617:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.545632:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5469, 5469] 00020000:00200000:2.0:1789760721.545641:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22401024 chunk: [22401024, 22405120) 22405120, 4096 00020000:00200000:2.0:1789760721.545646:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.545665:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5469, 5469] 00000008:00010000:2.0:1789760721.545684:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.545690:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.545772:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22401024, 22405120) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.545787:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22401024=0x155d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.545850:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.545853:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22405120 00000020:00010000:2.0:1789760721.545861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.545864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.545867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.545871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.545873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.545881:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.545888:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.545905:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760721.546141:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 22401024<>22405120 00000100:00100000:2.0:1789760721.546194:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760721.546215:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.617309:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972788 gen 3 00000008:00010000:3.0:1789760721.617361:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760721.617369:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.617380:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.617435:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760721.617465:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.617567:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5469, 5469] 00000080:00200000:2.0:1789760721.617584:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.617590:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.617604:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22405120 i_blocks 45063 00000020:00200000:2.0:1789760721.617608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.617617:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.617652:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22405120, count: 4096 00000080:00200000:2.0:1789760721.617659:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22405120, bytes: 4096 00000080:00200000:2.0:1789760721.617668:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.617676:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.617679:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22405120, 22409216) 0 0 00000080:00200000:2.0:1789760721.617683:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.617765:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5470, 5470] 00020000:00200000:2.0:1789760721.617775:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22405120 chunk: [22405120, 22409216) 22409216, 4096 00020000:00200000:2.0:1789760721.617779:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.617798:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5470, 5470] 00000008:00010000:2.0:1789760721.617815:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.617821:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.617833:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22405120, 22409216) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.617847:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22405120=0x155e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.617903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.617906:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22409216 00000020:00010000:2.0:1789760721.617915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.617918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.617921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.617924:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.617927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.617935:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.617941:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.618072:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760721.618176:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 22405120<>22409216 00000100:00100000:3.0:1789760721.618301:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760721.618320:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681280:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.684224:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972789 gen 3 00000008:00010000:3.0:1789760721.684279:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760721.684288:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.684301:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.684322:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760721.684352:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.684673:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5470, 5470] 00000080:00200000:2.0:1789760721.684689:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.684695:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.684711:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22409216 i_blocks 45063 00000020:00200000:2.0:1789760721.684714:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.684725:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.684759:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22409216, count: 4096 00000080:00200000:2.0:1789760721.684766:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22409216, bytes: 4096 00000080:00200000:2.0:1789760721.684776:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.684785:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.684788:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22409216, 22413312) 0 0 00000080:00200000:2.0:1789760721.684793:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.684808:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5471, 5471] 00020000:00200000:2.0:1789760721.684816:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22409216 chunk: [22409216, 22413312) 22413312, 4096 00020000:00200000:2.0:1789760721.684821:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.684840:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5471, 5471] 00000008:00010000:2.0:1789760721.684857:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.684864:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.684875:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22409216, 22413312) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.684888:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22409216=0x155f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.684946:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.684950:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22413312 00000020:00010000:2.0:1789760721.685015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.685020:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.685024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.685028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.685032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.685040:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.685047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.685064:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760721.685163:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22409216<>22413312 00000100:00100000:3.0:1789760721.685229:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.734818:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972790 gen 3 00000008:00010000:3.0:1789760721.734867:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760721.734875:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.734886:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.734905:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760721.734932:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.735665:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5471, 5471] 00000080:00200000:2.0:1789760721.735684:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.735691:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.735707:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22413312 i_blocks 45063 00000020:00200000:2.0:1789760721.735712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.735724:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.735759:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22413312, count: 4096 00000080:00200000:2.0:1789760721.735766:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22413312, bytes: 4096 00000080:00200000:2.0:1789760721.735778:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.735788:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.735791:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22413312, 22417408) 0 0 00000080:00200000:2.0:1789760721.735796:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.735813:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5472, 5472] 00020000:00200000:2.0:1789760721.735821:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22413312 chunk: [22413312, 22417408) 22417408, 4096 00020000:00200000:2.0:1789760721.735825:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.735846:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5472, 5472] 00000008:00010000:2.0:1789760721.735867:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.735873:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.735885:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22413312, 22417408) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.735900:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22413312=0x1560000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.735973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.735976:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22417408 00000020:00010000:2.0:1789760721.735984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.735988:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.735991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.735995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.735998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.736006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.736014:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.736029:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760721.736138:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 22413312<>22417408 00000100:00100000:3.0:1789760721.736376:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681536:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760721.786367:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760721.786384:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.786440:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.786462:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760721.786512:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.787428:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5472, 5472] 00000080:00200000:2.0:1789760721.787449:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.787456:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.787491:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22417408 i_blocks 45063 00000020:00200000:2.0:1789760721.787497:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.787510:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.787554:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22417408, count: 4096 00000080:00200000:2.0:1789760721.787562:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22417408, bytes: 4096 00000080:00200000:2.0:1789760721.787576:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.787585:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.787588:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22417408, 22421504) 0 0 00000080:00200000:2.0:1789760721.787592:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.787610:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5473, 5473] 00020000:00200000:2.0:1789760721.787618:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22417408 chunk: [22417408, 22421504) 22421504, 4096 00020000:00200000:2.0:1789760721.787622:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.787642:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5473, 5473] 00000008:00010000:2.0:1789760721.787660:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.787667:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.787679:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22417408, 22421504) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.787694:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22417408=0x1561000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.787757:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.787760:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22421504 00000020:00010000:2.0:1789760721.787768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.787772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.787775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.787779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.787782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.787790:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.787796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.787814:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760721.787917:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 22417408<>22421504 00000100:00100000:3.0:1789760721.788244:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.823489:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972792 gen 3 00000008:00010000:3.0:1789760721.823557:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760721.823566:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.823577:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.823598:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760721.823627:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681664:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.823710:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5473, 5473] 00000080:00200000:2.0:1789760721.823727:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.823734:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.823749:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22421504 i_blocks 45063 00000020:00200000:2.0:1789760721.823753:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.823763:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.823797:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22421504, count: 4096 00000080:00200000:2.0:1789760721.823806:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22421504, bytes: 4096 00000080:00200000:2.0:1789760721.823825:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.823834:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.823837:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22421504, 22425600) 0 0 00000080:00200000:2.0:1789760721.823842:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.823858:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5474, 5474] 00020000:00200000:2.0:1789760721.823868:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22421504 chunk: [22421504, 22425600) 22425600, 4096 00020000:00200000:2.0:1789760721.823873:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.823896:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5474, 5474] 00000008:00010000:2.0:1789760721.823917:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.823923:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.823937:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22421504, 22425600) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.823952:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22421504=0x1562000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.824028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.824031:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22425600 00000020:00010000:2.0:1789760721.824041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.824045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.824048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.824052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.824055:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.824062:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.824070:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.824087:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760721.824197:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 22421504<>22425600 00000100:00100000:3.0:1789760721.824487:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681792:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.855636:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972793 gen 3 00000008:00010000:3.0:1789760721.855687:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760721.855695:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.855705:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.855751:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760721.855774:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493681792:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.855850:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5474, 5474] 00000080:00200000:2.0:1789760721.855867:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.855873:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.855889:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22425600 i_blocks 45063 00000020:00200000:2.0:1789760721.855892:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.855904:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.855940:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22425600, count: 4096 00000080:00200000:2.0:1789760721.855947:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22425600, bytes: 4096 00000080:00200000:2.0:1789760721.855956:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.855972:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.855976:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22425600, 22429696) 0 0 00000080:00200000:2.0:1789760721.855981:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.855998:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5475, 5475] 00020000:00200000:2.0:1789760721.856007:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22425600 chunk: [22425600, 22429696) 22429696, 4096 00020000:00200000:2.0:1789760721.856012:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.856031:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5475, 5475] 00000008:00010000:2.0:1789760721.856046:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.856050:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.856061:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22425600, 22429696) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.856074:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22425600=0x1563000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.856127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.856130:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22429696 00000020:00010000:2.0:1789760721.856137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.856140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.856142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.856144:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.856146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.856152:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.856159:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.856170:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760721.856267:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 22425600<>22429696 00000100:00100000:3.0:1789760721.856331:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.891551:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972794 gen 3 00000008:00010000:3.0:1789760721.891712:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760721.891722:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.891733:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.891790:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760721.891832:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493681920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760721.892929:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5475, 5475] 00000080:00200000:2.0:1789760721.892948:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.892955:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.892971:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22429696 i_blocks 45063 00000020:00200000:2.0:1789760721.892976:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.892988:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.893022:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22429696, count: 4096 00000080:00200000:2.0:1789760721.893030:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22429696, bytes: 4096 00000080:00200000:2.0:1789760721.893041:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.893050:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.893054:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22429696, 22433792) 0 0 00000080:00200000:2.0:1789760721.893059:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.893077:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5476, 5476] 00020000:00200000:2.0:1789760721.893086:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22429696 chunk: [22429696, 22433792) 22433792, 4096 00020000:00200000:2.0:1789760721.893091:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.893110:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5476, 5476] 00000008:00010000:2.0:1789760721.893129:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.893136:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760721.893148:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22429696, 22433792) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.893162:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22429696=0x1564000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.893229:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.893232:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22433792 00000020:00010000:2.0:1789760721.893240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.893244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.893247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.893251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.893254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.893262:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.893269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.897026:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760721.897148:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 22429696<>22433792 00000100:00100000:2.0:1789760721.897204:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493682048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760721.950356:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972795 gen 3 00000008:00010000:3.0:1789760721.952778:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760721.952794:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.952809:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.953024:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760721.953060:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493682048:192.168.203.160@tcp:4:dd.0 00000020:00010000:1.0:1789760722.168075:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760722.168078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760722.168091:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760722.168099:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760722.168122:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760722.168237:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 22450176<>22454272 00000100:00100000:1.0:1789760722.168297:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760722.168322:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493682688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.215193:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972800 gen 3 00000008:00010000:1.0:1789760722.215246:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760722.215255:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.215266:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.215292:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760722.215322:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493682688:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760722.215439:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5481, 5481] 00000080:00200000:3.0:1789760722.215457:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.215464:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.215489:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22454272 i_blocks 45063 00000020:00200000:3.0:1789760722.215494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.215505:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.215553:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22454272, count: 4096 00000080:00200000:3.0:1789760722.215560:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22454272, bytes: 4096 00000080:00200000:3.0:1789760722.215569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.215579:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.215581:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22454272, 22458368) 0 0 00000080:00200000:3.0:1789760722.215586:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.215603:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5482, 5482] 00020000:00200000:3.0:1789760722.215611:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22454272 chunk: [22454272, 22458368) 22458368, 4096 00020000:00200000:3.0:1789760722.215616:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.215637:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5482, 5482] 00000008:00010000:3.0:1789760722.215656:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.215663:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760722.215673:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22454272, 22458368) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.215687:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22454272=0x156a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.215745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.215748:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22458368 00000020:00010000:3.0:1789760722.215756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.215760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.215763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.215766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.215769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.215777:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.215784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.215800:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760722.216156:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22454272<>22458368 00000100:00100000:3.0:1789760722.216213:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493682816:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760722.261463:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972801 gen 3 00000008:00010000:3.0:1789760722.261529:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760722.261602:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.261613:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.261687:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760722.261717:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493682816:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760722.261799:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5482, 5482] 00000080:00200000:1.0:1789760722.262097:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760722.262106:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760722.262127:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22458368 i_blocks 45063 00000020:00200000:1.0:1789760722.262131:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760722.262226:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760722.262272:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22458368, count: 4096 00000080:00200000:1.0:1789760722.262281:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22458368, bytes: 4096 00000080:00200000:1.0:1789760722.262293:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760722.262303:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760722.262305:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22458368, 22462464) 0 0 00000080:00200000:1.0:1789760722.262310:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760722.262328:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5483, 5483] 00020000:00200000:1.0:1789760722.262338:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22458368 chunk: [22458368, 22462464) 22462464, 4096 00020000:00200000:1.0:1789760722.262343:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760722.262363:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5483, 5483] 00000008:00010000:1.0:1789760722.262381:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760722.262388:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760722.262428:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22458368, 22462464) append 0 append lockpos 0 00000080:00200000:1.0:1789760722.262444:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22458368=0x156b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760722.262520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760722.262523:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22462464 00000020:00010000:1.0:1789760722.262598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:1.0:1789760722.262603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760722.262607:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760722.262610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760722.262614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760722.262625:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760722.262632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760722.262652:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760722.262761:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 22458368<>22462464 00000080:00200000:2.0:1789760722.309079:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5483, 5483] 00000080:00200000:2.0:1789760722.309107:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.309114:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760722.309131:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22462464 i_blocks 45063 00000020:00200000:2.0:1789760722.309137:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760722.309154:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760722.309191:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22462464, count: 4096 00000080:00200000:2.0:1789760722.309200:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22462464, bytes: 4096 00000080:00200000:2.0:1789760722.309215:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760722.309224:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760722.309226:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22462464, 22466560) 0 0 00000080:00200000:2.0:1789760722.309231:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760722.309248:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5484, 5484] 00020000:00200000:2.0:1789760722.309257:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22462464 chunk: [22462464, 22466560) 22466560, 4096 00020000:00200000:2.0:1789760722.309260:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760722.309279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5484, 5484] 00000008:00010000:2.0:1789760722.309297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760722.309303:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760722.309314:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22462464, 22466560) append 0 append lockpos 0 00000080:00200000:2.0:1789760722.309329:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22462464=0x156c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760722.309426:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760722.309429:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22466560 00000020:00010000:2.0:1789760722.309438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760722.309442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.309444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.309448:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760722.309451:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760722.309460:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760722.309466:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760722.309501:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760722.309610:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 22462464<>22466560 00000100:00100000:3.0:1789760722.310799:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683072:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760722.352469:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760722.352554:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.352567:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.352626:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760722.352655:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760722.352730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5484, 5484] 00000080:00200000:2.0:1789760722.352744:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.352750:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.352845:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22466560 i_blocks 45063 00000020:00200000:3.0:1789760722.352849:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.352860:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.352896:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22466560, count: 4096 00000080:00200000:3.0:1789760722.352903:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22466560, bytes: 4096 00000080:00200000:3.0:1789760722.352913:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.352922:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.352925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22466560, 22470656) 0 0 00000080:00200000:3.0:1789760722.352929:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.352945:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5485, 5485] 00020000:00200000:3.0:1789760722.353197:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22466560 chunk: [22466560, 22470656) 22470656, 4096 00020000:00200000:3.0:1789760722.353203:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.353226:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5485, 5485] 00000008:00010000:3.0:1789760722.353248:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.353254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760722.353266:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22466560, 22470656) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.353385:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22466560=0x156d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.353639:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.353642:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22470656 00000020:00010000:3.0:1789760722.353652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.353656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.353660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.353664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.353667:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.353675:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.353683:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.353701:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760722.353809:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22466560<>22470656 00000100:00100000:3.0:1789760722.353870:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760722.353891:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760722.391512:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972804 gen 3 00000008:00010000:3.0:1789760722.391628:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760722.391638:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.391651:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.391675:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760722.391705:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683200:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760722.436751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5486, 5486] 00000080:00200000:3.0:1789760722.436768:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.436775:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.436792:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22474752 i_blocks 45063 00000020:00200000:3.0:1789760722.436796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.436806:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.437029:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22474752, count: 4096 00000080:00200000:3.0:1789760722.437039:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22474752, bytes: 4096 00000080:00200000:3.0:1789760722.437210:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.437220:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.437222:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22474752, 22478848) 0 0 00000080:00200000:3.0:1789760722.437227:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.437244:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5487, 5487] 00020000:00200000:3.0:1789760722.437253:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22474752 chunk: [22474752, 22478848) 22478848, 4096 00020000:00200000:3.0:1789760722.437257:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.437281:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5487, 5487] 00000008:00010000:3.0:1789760722.437302:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.437309:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760722.437321:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22474752, 22478848) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.437337:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22474752=0x156f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.437513:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.437516:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22478848 00000020:00010000:3.0:1789760722.437591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.437596:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.437599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.437603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.437606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.437616:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.437623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.437645:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760722.437764:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22474752<>22478848 00000100:00100000:2.0:1789760722.438290:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683456:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760722.485680:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972806 gen 3 00000008:00010000:3.0:1789760722.485732:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760722.485740:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.485751:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.485835:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760722.485865:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493683456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.486766:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493683584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.523905:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972807 gen 3 00000008:00010000:1.0:1789760722.523986:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760722.523996:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.524007:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.524027:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760722.524062:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493683584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.525133:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683712:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.562386:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972808 gen 3 00000008:00010000:1.0:1789760722.562469:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760722.562486:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.562497:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.562520:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760722.562550:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683712:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.563279:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.563295:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683840:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.592388:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972809 gen 3 00000008:00010000:1.0:1789760722.592482:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760722.592492:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.592503:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.592530:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760722.592559:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683840:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.593353:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683968:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.652346:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972810 gen 3 00000008:00010000:1.0:1789760722.652432:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760722.652442:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.652454:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.652649:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760722.652682:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493683968:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.655420:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.655453:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684096:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.695122:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972811 gen 3 00000008:00010000:1.0:1789760722.695173:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760722.695181:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.695192:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.695217:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760722.695249:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684096:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.696434:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684224:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.733503:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972812 gen 3 00000008:00010000:1.0:1789760722.733554:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760722.733563:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.733575:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.733601:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760722.733630:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684224:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.734685:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.734707:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684352:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.773802:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972813 gen 3 00000008:00010000:1.0:1789760722.773864:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760722.773872:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.773884:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.773907:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760722.773936:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684352:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.775015:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684480:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.804174:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972814 gen 3 00000008:00010000:1.0:1789760722.804224:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760722.804234:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.804244:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.804269:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760722.804297:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684480:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.805049:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.805065:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684608:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.862142:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972815 gen 3 00000008:00010000:1.0:1789760722.862191:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760722.862200:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.862211:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.862233:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760722.862261:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684608:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.863310:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684736:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.902131:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972816 gen 3 00000008:00010000:1.0:1789760722.902225:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760722.902233:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.902243:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.902306:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760722.902337:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684736:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760722.903556:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.903579:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684864:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760722.953044:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760722.953058:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.953069:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.953092:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760722.953120:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493684864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760722.987971:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5499, 5499] 00000080:00200000:3.0:1789760722.987989:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.987994:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.988011:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22528000 i_blocks 45063 00000020:00200000:3.0:1789760722.988015:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.988026:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.988061:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22528000, count: 4096 00000080:00200000:3.0:1789760722.988068:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22528000, bytes: 4096 00000080:00200000:3.0:1789760722.988079:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.988087:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.988090:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22528000, 22532096) 0 0 00000080:00200000:3.0:1789760722.988094:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.988110:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5500, 5500] 00020000:00200000:3.0:1789760722.988118:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22528000 chunk: [22528000, 22532096) 22532096, 4096 00020000:00200000:3.0:1789760722.988122:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.988140:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5500, 5500] 00000008:00010000:3.0:1789760722.988157:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.988163:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760722.988174:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22528000, 22532096) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.988188:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22528000=0x157c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.988362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.988366:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22532096 00000020:00010000:3.0:1789760722.988374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.988379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.988382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.988386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.988388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.988422:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.988430:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.988445:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760722.988623:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22528000<>22532096 00000100:00100000:3.0:1789760722.988667:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760722.988686:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493685120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.019385:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972819 gen 3 00000008:00010000:3.0:1789760723.019462:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760723.019480:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.019492:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.019515:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760723.019543:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493685120:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.020314:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493685248:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760723.054914:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760723.054932:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.054943:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.055061:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760723.055092:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493685248:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760723.056864:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5501, 5501] 00000080:00200000:3.0:1789760723.056886:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760723.056893:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760723.056913:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22536192 i_blocks 45063 00000020:00200000:3.0:1789760723.056918:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760723.056933:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760723.056966:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22536192, count: 4096 00000080:00200000:3.0:1789760723.056974:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22536192, bytes: 4096 00000080:00200000:3.0:1789760723.056988:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760723.056997:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760723.057000:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22536192, 22540288) 0 0 00000080:00200000:3.0:1789760723.057004:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760723.057021:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5502, 5502] 00020000:00200000:3.0:1789760723.057030:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22536192 chunk: [22536192, 22540288) 22540288, 4096 00020000:00200000:3.0:1789760723.057035:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760723.057056:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5502, 5502] 00000008:00010000:3.0:1789760723.057074:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760723.057081:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760723.057093:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22536192, 22540288) append 0 append lockpos 0 00000080:00200000:3.0:1789760723.057108:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22536192=0x157e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760723.057171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760723.057175:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22540288 00000020:00010000:3.0:1789760723.057243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760723.057248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.057252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.057256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760723.057258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760723.057266:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760723.057274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760723.057300:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760723.057445:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22536192<>22540288 00000100:00100000:2.0:1789760723.057559:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760723.057587:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493685376:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.118281:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972821 gen 3 00000008:00010000:3.0:1789760723.118337:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760723.118343:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.118350:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.118367:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760723.118391:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493685376:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.118535:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5502, 5502] 00000080:00200000:2.0:1789760723.118549:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.118553:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.118564:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22540288 i_blocks 45063 00000020:00200000:2.0:1789760723.118568:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.118576:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.118605:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22540288, count: 4096 00000080:00200000:2.0:1789760723.118611:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22540288, bytes: 4096 00000080:00200000:2.0:1789760723.118620:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.118627:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.118630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22540288, 22544384) 0 0 00000080:00200000:2.0:1789760723.118633:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.118645:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5503, 5503] 00020000:00200000:2.0:1789760723.118651:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22540288 chunk: [22540288, 22544384) 22544384, 4096 00020000:00200000:2.0:1789760723.118655:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.118670:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5503, 5503] 00000008:00010000:2.0:1789760723.118683:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.118687:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.118696:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22540288, 22544384) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.118706:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22540288=0x157f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.118748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.118751:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22544384 00000020:00010000:2.0:1789760723.118757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.118760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.118763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.118766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.118769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.118775:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.118780:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.118795:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760723.118869:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 22540288<>22544384 00000100:00100000:3.0:1789760723.119108:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493685504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.164218:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972822 gen 3 00000008:00010000:3.0:1789760723.164249:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760723.164255:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.164263:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.164287:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760723.164311:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493685504:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.164994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493685632:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.217602:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972823 gen 3 00000008:00010000:1.0:1789760723.217662:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760723.217671:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.217683:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.217774:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760723.217806:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493685632:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.219224:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493685760:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.290030:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972824 gen 3 00000008:00010000:1.0:1789760723.290080:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760723.290089:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.290102:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.290172:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760723.290203:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493685760:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.290758:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5505, 5505] 00000080:00200000:2.0:1789760723.290775:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.290781:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.290797:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22552576 i_blocks 45063 00000020:00200000:2.0:1789760723.290800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.290811:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.290843:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22552576, count: 4096 00000080:00200000:2.0:1789760723.290850:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22552576, bytes: 4096 00000080:00200000:2.0:1789760723.290926:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.290935:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.290937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22552576, 22556672) 0 0 00000080:00200000:2.0:1789760723.290942:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.290962:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5506, 5506] 00020000:00200000:2.0:1789760723.290970:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22552576 chunk: [22552576, 22556672) 22556672, 4096 00020000:00200000:2.0:1789760723.290975:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.291070:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5506, 5506] 00000008:00010000:2.0:1789760723.291091:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.291097:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.291110:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22552576, 22556672) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.291124:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22552576=0x1582000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.291263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.291266:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22556672 00000020:00010000:2.0:1789760723.291275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.291279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.291282:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.291285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.291288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.291297:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.291304:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.291325:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760723.291492:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 22552576<>22556672 00000100:00100000:3.0:1789760723.291584:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493685888:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.350330:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972825 gen 3 00000008:00010000:3.0:1789760723.350382:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760723.350391:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.350430:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.350452:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760723.350490:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493685888:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.351367:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493686016:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.420359:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972826 gen 3 00000008:00010000:1.0:1789760723.420446:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760723.420455:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.420478:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.420501:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760723.420530:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493686016:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.421163:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493686144:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760723.522443:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972827 gen 3 00000008:00010000:1.0:1789760723.522522:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760723.522531:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.522542:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.522571:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760723.522600:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493686144:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760723.523073:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5508, 5508] 00000080:00200000:3.0:1789760723.523090:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760723.523096:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760723.523112:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22564864 i_blocks 45063 00000020:00200000:3.0:1789760723.523115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760723.523125:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760723.523164:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22564864, count: 4096 00000080:00200000:3.0:1789760723.523172:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22564864, bytes: 4096 00000080:00200000:3.0:1789760723.523187:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760723.523196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760723.523198:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22564864, 22568960) 0 0 00000080:00200000:3.0:1789760723.523203:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760723.523222:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5509, 5509] 00020000:00200000:3.0:1789760723.524137:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22564864 chunk: [22564864, 22568960) 22568960, 4096 00020000:00200000:3.0:1789760723.524146:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760723.524171:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5509, 5509] 00000008:00010000:3.0:1789760723.524193:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760723.524199:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760723.524214:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22564864, 22568960) append 0 append lockpos 0 00000080:00200000:3.0:1789760723.524231:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22564864=0x1585000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760723.524300:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760723.524304:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22568960 00000020:00010000:3.0:1789760723.524315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760723.524318:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.524321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.524325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760723.524329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760723.524339:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760723.524347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760723.524367:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760723.524517:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22564864<>22568960 00000100:00100000:3.0:1789760723.524564:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760723.524589:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686272:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760723.604231:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760723.604242:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.604251:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.604274:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760723.604300:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.609192:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5509, 5509] 00000080:00200000:2.0:1789760723.609213:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.609219:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.609233:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22568960 i_blocks 45063 00000020:00200000:2.0:1789760723.609238:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.609249:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.609277:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22568960, count: 4096 00000080:00200000:2.0:1789760723.609284:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22568960, bytes: 4096 00000080:00200000:2.0:1789760723.609295:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.609302:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.609304:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22568960, 22573056) 0 0 00000080:00200000:2.0:1789760723.609308:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.609321:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5510, 5510] 00020000:00200000:2.0:1789760723.609328:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22568960 chunk: [22568960, 22573056) 22573056, 4096 00020000:00200000:2.0:1789760723.609332:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.609350:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5510, 5510] 00000008:00010000:2.0:1789760723.609364:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.609369:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.609378:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22568960, 22573056) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.609390:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22568960=0x1586000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.609458:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.609461:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22573056 00000020:00010000:2.0:1789760723.609476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.609480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.609482:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.609485:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.609488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.609494:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.609499:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.609515:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760723.609771:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22568960<>22573056 00000100:00100000:3.0:1789760723.609843:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.746345:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972829 gen 3 00000008:00010000:3.0:1789760723.746442:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760723.746449:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.746458:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.746485:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760723.746510:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686400:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.746813:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5510, 5510] 00000080:00200000:2.0:1789760723.746827:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.746833:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.746845:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22573056 i_blocks 45063 00000020:00200000:2.0:1789760723.746848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.746858:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.746885:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22573056, count: 4096 00000080:00200000:2.0:1789760723.746892:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22573056, bytes: 4096 00000080:00200000:2.0:1789760723.746901:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.746908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.746910:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22573056, 22577152) 0 0 00000080:00200000:2.0:1789760723.746915:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.746927:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5511, 5511] 00020000:00200000:2.0:1789760723.746934:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22573056 chunk: [22573056, 22577152) 22577152, 4096 00020000:00200000:2.0:1789760723.746938:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.746966:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5511, 5511] 00000008:00010000:2.0:1789760723.746981:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.746986:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.746996:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22573056, 22577152) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.747008:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22573056=0x1587000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.747052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.747055:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22577152 00000020:00010000:2.0:1789760723.747061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.747065:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.747068:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.747071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.747073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.747080:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.747086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.747102:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760723.747179:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22573056<>22577152 00000100:00100000:3.0:1789760723.747735:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.827636:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972830 gen 3 00000008:00010000:3.0:1789760723.827668:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760723.827675:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.827683:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.827702:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760723.827726:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686528:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.827813:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5511, 5511] 00000080:00200000:2.0:1789760723.827832:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.827838:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.827849:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22577152 i_blocks 45063 00000020:00200000:2.0:1789760723.827852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.827860:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.827886:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22577152, count: 4096 00000080:00200000:2.0:1789760723.827892:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22577152, bytes: 4096 00000080:00200000:2.0:1789760723.827900:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.827906:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.827908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22577152, 22581248) 0 0 00000080:00200000:2.0:1789760723.827911:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.827924:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5512, 5512] 00020000:00200000:2.0:1789760723.827930:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22577152 chunk: [22577152, 22581248) 22581248, 4096 00020000:00200000:2.0:1789760723.827934:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.827957:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5512, 5512] 00000008:00010000:2.0:1789760723.827970:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.827975:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.827984:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22577152, 22581248) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.827994:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22577152=0x1588000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.828274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.828277:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22581248 00000020:00010000:2.0:1789760723.828284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.828288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.828290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.828293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.828295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.828302:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.828308:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.828325:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760723.828681:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22577152<>22581248 00000100:00100000:3.0:1789760723.829154:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760723.829174:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760723.897513:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972831 gen 3 00000008:00010000:2.0:1789760723.897800:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:2.0:1789760723.897807:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760723.897816:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760723.897835:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:2.0:1789760723.897857:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686656:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760723.897940:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5512, 5512] 00000080:00200000:3.0:1789760723.898010:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760723.898015:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760723.898026:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22581248 i_blocks 45063 00000020:00200000:3.0:1789760723.898029:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760723.898037:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760723.898063:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22581248, count: 4096 00000080:00200000:3.0:1789760723.898069:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22581248, bytes: 4096 00000080:00200000:3.0:1789760723.898076:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760723.898083:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760723.898085:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22581248, 22585344) 0 0 00000080:00200000:3.0:1789760723.898088:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760723.898100:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5513, 5513] 00020000:00200000:3.0:1789760723.898106:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22581248 chunk: [22581248, 22585344) 22585344, 4096 00020000:00200000:3.0:1789760723.898110:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760723.898125:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5513, 5513] 00000008:00010000:3.0:1789760723.898137:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760723.898142:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760723.898151:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22581248, 22585344) append 0 append lockpos 0 00000080:00200000:3.0:1789760723.898161:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22581248=0x1589000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760723.898204:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760723.898207:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22585344 00000020:00010000:3.0:1789760723.898213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760723.898216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.898218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760723.898221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760723.898224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760723.898231:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760723.898236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760723.898251:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760723.898327:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22581248<>22585344 00000100:00100000:2.0:1789760723.901504:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686784:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760723.963261:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760723.963271:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.963281:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.963300:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760723.963325:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493686784:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760723.963416:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5513, 5513] 00000080:00200000:2.0:1789760723.963429:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.963434:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.963445:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22585344 i_blocks 45063 00000020:00200000:2.0:1789760723.963449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.963458:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.963498:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22585344, count: 4096 00000080:00200000:2.0:1789760723.963504:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22585344, bytes: 4096 00000080:00200000:2.0:1789760723.963512:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.963585:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.963588:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22585344, 22589440) 0 0 00000080:00200000:2.0:1789760723.963592:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.963605:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5514, 5514] 00020000:00200000:2.0:1789760723.963611:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22585344 chunk: [22585344, 22589440) 22589440, 4096 00020000:00200000:2.0:1789760723.963615:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.963631:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5514, 5514] 00000008:00010000:2.0:1789760723.963646:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.963650:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760723.963659:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22585344, 22589440) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.963670:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22585344=0x158a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.963709:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.963712:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22589440 00000020:00010000:2.0:1789760723.963718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.963722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.963725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.963728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.963730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.963753:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.963759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.963773:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760723.963864:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22585344<>22589440 00000100:00100000:3.0:1789760723.963975:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.049761:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972832 gen 3 00000100:00100000:3.0:1789760724.049793:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80380 x1876699493686912/t4294972833(4294972833) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760739 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760724.049830:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760724.049835:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.049843:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.049860:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760724.049880:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493686912:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.050149:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5514, 5514] 00000080:00200000:2.0:1789760724.050162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.050167:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.050178:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22589440 i_blocks 45063 00000020:00200000:2.0:1789760724.050182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.050190:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.050218:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22589440, count: 4096 00000080:00200000:2.0:1789760724.050224:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22589440, bytes: 4096 00000080:00200000:2.0:1789760724.050233:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.050240:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.050242:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22589440, 22593536) 0 0 00000080:00200000:2.0:1789760724.050246:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.050258:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5515, 5515] 00020000:00200000:2.0:1789760724.050266:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22589440 chunk: [22589440, 22593536) 22593536, 4096 00020000:00200000:2.0:1789760724.050269:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.050284:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5515, 5515] 00000008:00010000:2.0:1789760724.050297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.050302:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.050311:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22589440, 22593536) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.050323:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22589440=0x158b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.050367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.050369:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22593536 00000020:00010000:2.0:1789760724.050376:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.050379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.050382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.050385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.050388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.050447:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.050453:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.050480:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760724.050558:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 22589440<>22593536 00000100:00100000:3.0:1789760724.050814:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760724.050829:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.087843:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972833 gen 3 00000100:00100000:3.0:1789760724.087872:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80000 x1876699493687040/t4294972834(4294972834) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760740 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760724.087909:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760724.087914:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.087923:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.087952:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760724.087973:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687040:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.088148:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5515, 5515] 00000080:00200000:2.0:1789760724.088162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.088167:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.088179:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22593536 i_blocks 45063 00000020:00200000:2.0:1789760724.088182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.088192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.088215:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22593536, count: 4096 00000080:00200000:2.0:1789760724.088221:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22593536, bytes: 4096 00000080:00200000:2.0:1789760724.088228:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.088235:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.088237:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22593536, 22597632) 0 0 00000080:00200000:2.0:1789760724.088241:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.088254:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5516, 5516] 00020000:00200000:2.0:1789760724.088260:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22593536 chunk: [22593536, 22597632) 22597632, 4096 00020000:00200000:2.0:1789760724.088264:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.088278:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5516, 5516] 00000008:00010000:2.0:1789760724.088290:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.088295:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.088304:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22593536, 22597632) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.088314:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22593536=0x158c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.088355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.088357:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22597632 00000020:00010000:2.0:1789760724.088364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.088367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.088369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.088372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.088375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.088381:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.088386:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.088449:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760724.088536:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22593536<>22597632 00000100:00100000:3.0:1789760724.089101:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760724.089119:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.144281:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972835 gen 3 00000008:00010000:3.0:1789760724.144321:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760724.144327:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.144334:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.144348:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760724.144368:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687168:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.144556:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5516, 5516] 00000080:00200000:2.0:1789760724.144565:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.144569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.144580:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22597632 i_blocks 45063 00000020:00200000:2.0:1789760724.144584:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.144594:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.144623:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22597632, count: 4096 00000080:00200000:2.0:1789760724.144629:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22597632, bytes: 4096 00000080:00200000:2.0:1789760724.144637:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.144644:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.144646:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22597632, 22601728) 0 0 00000080:00200000:2.0:1789760724.144650:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.144662:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5517, 5517] 00020000:00200000:2.0:1789760724.144669:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22597632 chunk: [22597632, 22601728) 22601728, 4096 00020000:00200000:2.0:1789760724.144672:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.144687:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5517, 5517] 00000008:00010000:2.0:1789760724.144700:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.144705:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.144714:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22597632, 22601728) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.144724:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22597632=0x158d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.144766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.144768:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22601728 00000020:00010000:2.0:1789760724.144775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.144778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.144781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.144784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.144787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.144792:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.144798:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.144812:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760724.144884:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 22597632<>22601728 00000100:00100000:3.0:1789760724.146031:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760724.146052:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687296:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760724.207004:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760724.227552:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972836 gen 3 00000008:00010000:3.0:1789760724.227590:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760724.227596:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.227606:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.227675:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760724.227700:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.229968:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5517, 5517] 00000080:00200000:2.0:1789760724.229988:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.229993:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.230006:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22601728 i_blocks 45063 00000020:00200000:2.0:1789760724.230010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.230022:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.230055:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22601728, count: 4096 00000080:00200000:2.0:1789760724.230062:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22601728, bytes: 4096 00000080:00200000:2.0:1789760724.230072:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.230079:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.230081:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22601728, 22605824) 0 0 00000080:00200000:2.0:1789760724.230084:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.230097:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5518, 5518] 00020000:00200000:2.0:1789760724.230103:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22601728 chunk: [22601728, 22605824) 22605824, 4096 00020000:00200000:2.0:1789760724.230108:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.230124:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5518, 5518] 00000008:00010000:2.0:1789760724.230137:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.230141:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.230150:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22601728, 22605824) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.230160:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22601728=0x158e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.230214:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.230217:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22605824 00000020:00010000:2.0:1789760724.230224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.230228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.230230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.230234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.230236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.230252:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.230258:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.230282:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760724.230363:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 22601728<>22605824 00000100:00100000:3.0:1789760724.230675:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760724.230697:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687424:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760724.289766:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760724.289932:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.290004:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.290022:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760724.290048:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.290226:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5518, 5518] 00000080:00200000:2.0:1789760724.290241:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.290246:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.290257:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22605824 i_blocks 45063 00000020:00200000:2.0:1789760724.290260:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.290269:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.290296:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22605824, count: 4096 00000080:00200000:2.0:1789760724.290303:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22605824, bytes: 4096 00000080:00200000:2.0:1789760724.290311:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.290318:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.290319:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22605824, 22609920) 0 0 00000080:00200000:2.0:1789760724.290323:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.290336:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5519, 5519] 00020000:00200000:2.0:1789760724.290344:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22605824 chunk: [22605824, 22609920) 22609920, 4096 00020000:00200000:2.0:1789760724.290348:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.290363:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5519, 5519] 00000008:00010000:2.0:1789760724.290376:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.290619:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.290632:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22605824, 22609920) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.290644:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22605824=0x158f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.290690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.290693:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22609920 00000020:00010000:2.0:1789760724.290699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.290703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.290705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.290708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.290711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.290717:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.290723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.290737:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760724.290815:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22605824<>22609920 00000100:00100000:3.0:1789760724.290870:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687552:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.341891:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972838 gen 3 00000008:00010000:3.0:1789760724.341938:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760724.342002:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.342011:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.342028:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760724.342055:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493687552:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.342256:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5519, 5519] 00000080:00200000:2.0:1789760724.342269:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.342274:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.342286:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22609920 i_blocks 45063 00000020:00200000:2.0:1789760724.342289:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.342298:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.342324:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22609920, count: 4096 00000080:00200000:2.0:1789760724.342330:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22609920, bytes: 4096 00000080:00200000:2.0:1789760724.342337:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.342344:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.342346:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22609920, 22614016) 0 0 00000080:00200000:2.0:1789760724.342349:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.342362:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5520, 5520] 00020000:00200000:2.0:1789760724.342369:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22609920 chunk: [22609920, 22614016) 22614016, 4096 00020000:00200000:2.0:1789760724.342373:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.342387:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5520, 5520] 00000008:00010000:2.0:1789760724.342429:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.342434:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.342443:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22609920, 22614016) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.342453:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22609920=0x1590000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.342504:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.342507:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22614016 00000020:00010000:2.0:1789760724.342520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.342525:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.342528:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.342531:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.342534:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.342541:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.342546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.342560:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760724.342630:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 22609920<>22614016 00000100:00100000:3.0:1789760724.342923:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.391238:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972839 gen 3 00000008:00010000:3.0:1789760724.391275:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760724.391281:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.391290:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.391310:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760724.391334:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.391435:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5520, 5520] 00000080:00200000:2.0:1789760724.391448:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.391452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.391478:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22614016 i_blocks 45063 00000020:00200000:2.0:1789760724.391482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.391492:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.391519:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22614016, count: 4096 00000080:00200000:2.0:1789760724.391525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22614016, bytes: 4096 00000080:00200000:2.0:1789760724.391534:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.391540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.391543:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22614016, 22618112) 0 0 00000080:00200000:2.0:1789760724.391546:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.391559:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5521, 5521] 00020000:00200000:2.0:1789760724.391566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22614016 chunk: [22614016, 22618112) 22618112, 4096 00020000:00200000:2.0:1789760724.391569:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.391584:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5521, 5521] 00000008:00010000:2.0:1789760724.391597:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.391602:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.391611:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22614016, 22618112) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.391621:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22614016=0x1591000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.391816:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.391819:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22618112 00000020:00010000:2.0:1789760724.391886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.391891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.391893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.391896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.391898:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.391904:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.391909:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.391924:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760724.392004:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22614016<>22618112 00000100:00100000:3.0:1789760724.392133:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760724.392145:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687808:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760724.450116:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760724.450128:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.450151:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.450169:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760724.450195:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.450381:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5521, 5521] 00000080:00200000:2.0:1789760724.450424:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.450430:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.450441:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22618112 i_blocks 45063 00000020:00200000:2.0:1789760724.450445:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.450454:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.450493:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22618112, count: 4096 00000080:00200000:2.0:1789760724.450499:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22618112, bytes: 4096 00000080:00200000:2.0:1789760724.450507:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.450513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.450515:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22618112, 22622208) 0 0 00000080:00200000:2.0:1789760724.450518:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.450531:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5522, 5522] 00020000:00200000:2.0:1789760724.450537:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22618112 chunk: [22618112, 22622208) 22622208, 4096 00020000:00200000:2.0:1789760724.450540:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.450556:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5522, 5522] 00000008:00010000:2.0:1789760724.450568:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.450573:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.450583:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22618112, 22622208) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.450592:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22618112=0x1592000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.450722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.450726:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22622208 00000020:00010000:2.0:1789760724.450732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.450736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.450739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.450742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.450744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.450945:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.450951:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.450966:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760724.451042:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22618112<>22622208 00000100:00100000:3.0:1789760724.451570:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687936:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.516280:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972841 gen 3 00000008:00010000:3.0:1789760724.516329:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760724.516335:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.516344:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.516361:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760724.516387:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493687936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.516773:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5522, 5522] 00000080:00200000:2.0:1789760724.516786:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.516791:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.516813:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22622208 i_blocks 45063 00000020:00200000:2.0:1789760724.517126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.517145:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.517172:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22622208, count: 4096 00000080:00200000:2.0:1789760724.517179:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22622208, bytes: 4096 00000080:00200000:2.0:1789760724.517189:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.517196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.517198:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22622208, 22626304) 0 0 00000080:00200000:2.0:1789760724.517202:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.517215:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5523, 5523] 00020000:00200000:2.0:1789760724.517223:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22622208 chunk: [22622208, 22626304) 22626304, 4096 00020000:00200000:2.0:1789760724.517227:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.517241:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5523, 5523] 00000008:00010000:2.0:1789760724.517255:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.517259:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.517268:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22622208, 22626304) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.517278:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22622208=0x1593000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.517322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.517325:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22626304 00000020:00010000:2.0:1789760724.517331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.517334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.517337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.517340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.517343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.517349:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.517354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.517370:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760724.517492:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22622208<>22626304 00000100:00100000:3.0:1789760724.518111:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688064:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760724.565138:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760724.565149:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.565158:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.565175:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760724.565200:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.565540:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5523, 5523] 00000080:00200000:2.0:1789760724.565553:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.565558:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.565569:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22626304 i_blocks 45063 00000020:00200000:2.0:1789760724.565572:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.565581:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.565609:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22626304, count: 4096 00000080:00200000:2.0:1789760724.565615:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22626304, bytes: 4096 00000080:00200000:2.0:1789760724.565623:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.565629:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.565631:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22626304, 22630400) 0 0 00000080:00200000:2.0:1789760724.565634:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.565646:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5524, 5524] 00020000:00200000:2.0:1789760724.565653:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22626304 chunk: [22626304, 22630400) 22630400, 4096 00020000:00200000:2.0:1789760724.565656:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.565671:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5524, 5524] 00000008:00010000:2.0:1789760724.565697:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.565702:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.565712:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22626304, 22630400) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.565724:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22626304=0x1594000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.565767:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.565770:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22630400 00000020:00010000:2.0:1789760724.565776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.565780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.565783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.565786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.565789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.565795:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.566062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.566079:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760724.566155:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 22626304<>22630400 00000100:00100000:3.0:1789760724.566268:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.639718:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972842 gen 3 00000100:00100000:3.0:1789760724.639747:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83800 x1876699493688192/t4294972843(4294972843) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760740 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760724.639793:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760724.639798:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.639806:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.639825:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760724.639847:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.640043:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5524, 5524] 00000080:00200000:2.0:1789760724.640054:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.640059:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.640069:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22630400 i_blocks 45063 00000020:00200000:2.0:1789760724.640072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.640080:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.640107:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22630400, count: 4096 00000080:00200000:2.0:1789760724.640112:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22630400, bytes: 4096 00000080:00200000:2.0:1789760724.640122:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.640128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.640130:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22630400, 22634496) 0 0 00000080:00200000:2.0:1789760724.640134:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.640147:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5525, 5525] 00020000:00200000:2.0:1789760724.640154:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22630400 chunk: [22630400, 22634496) 22634496, 4096 00020000:00200000:2.0:1789760724.640158:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.640172:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5525, 5525] 00000008:00010000:2.0:1789760724.640184:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.640188:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.640195:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22630400, 22634496) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.640206:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22630400=0x1595000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.640244:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.640247:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22634496 00000020:00010000:2.0:1789760724.640252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.640256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.640258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.640262:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.640264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.640269:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.640274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.640286:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760724.640357:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 22630400<>22634496 00000100:00100000:3.0:1789760724.640465:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.709533:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972844 gen 3 00000008:00010000:3.0:1789760724.709579:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760724.709585:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.709594:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.709615:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760724.709642:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688320:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.709727:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5525, 5525] 00000080:00200000:2.0:1789760724.709740:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.709746:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.709767:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22634496 i_blocks 45063 00000020:00200000:2.0:1789760724.709770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.709780:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.709806:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22634496, count: 4096 00000080:00200000:2.0:1789760724.709832:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22634496, bytes: 4096 00000080:00200000:2.0:1789760724.709840:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.709847:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.709848:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22634496, 22638592) 0 0 00000080:00200000:2.0:1789760724.709852:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.709865:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5526, 5526] 00020000:00200000:2.0:1789760724.709872:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22634496 chunk: [22634496, 22638592) 22638592, 4096 00020000:00200000:2.0:1789760724.709875:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.709900:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5526, 5526] 00000008:00010000:2.0:1789760724.709913:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.709918:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.709926:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22634496, 22638592) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.709948:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22634496=0x1596000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.709992:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.709995:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22638592 00000020:00010000:2.0:1789760724.710001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.710005:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.710008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.710011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.710014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.710020:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.710026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.710041:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760724.710293:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 22634496<>22638592 00000100:00100000:3.0:1789760724.710605:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.767292:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972845 gen 3 00000008:00010000:3.0:1789760724.767329:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760724.767336:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.767345:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.767613:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760724.767639:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688448:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.767750:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5526, 5526] 00000080:00200000:2.0:1789760724.767762:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.767767:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.767778:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22638592 i_blocks 45063 00000020:00200000:2.0:1789760724.767781:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.767802:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.767829:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22638592, count: 4096 00000080:00200000:2.0:1789760724.767835:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22638592, bytes: 4096 00000080:00200000:2.0:1789760724.767844:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.767850:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.767852:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22638592, 22642688) 0 0 00000080:00200000:2.0:1789760724.767856:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.767868:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5527, 5527] 00020000:00200000:2.0:1789760724.767874:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22638592 chunk: [22638592, 22642688) 22642688, 4096 00020000:00200000:2.0:1789760724.767878:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.767893:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5527, 5527] 00000008:00010000:2.0:1789760724.767906:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.767910:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.767919:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22638592, 22642688) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.767931:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22638592=0x1597000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.768227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.768230:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22642688 00000020:00010000:2.0:1789760724.768238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.768242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.768245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.768248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.768250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.768256:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.768262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.768278:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760724.768355:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 22638592<>22642688 00000100:00100000:3.0:1789760724.768481:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.828522:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972846 gen 3 00000008:00010000:3.0:1789760724.828558:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760724.828564:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.828572:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.828590:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760724.828615:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688576:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.828707:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5527, 5527] 00000080:00200000:2.0:1789760724.828722:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.828727:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.828738:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22642688 i_blocks 45063 00000020:00200000:2.0:1789760724.828741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.828749:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.828774:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22642688, count: 4096 00000080:00200000:2.0:1789760724.828779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22642688, bytes: 4096 00000080:00200000:2.0:1789760724.828787:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.828794:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.828796:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22642688, 22646784) 0 0 00000080:00200000:2.0:1789760724.828799:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.828812:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5528, 5528] 00020000:00200000:2.0:1789760724.828818:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22642688 chunk: [22642688, 22646784) 22646784, 4096 00020000:00200000:2.0:1789760724.828822:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.828837:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5528, 5528] 00000008:00010000:2.0:1789760724.828850:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.828854:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.828864:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22642688, 22646784) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.828874:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22642688=0x1598000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.828919:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.828922:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22646784 00000020:00010000:2.0:1789760724.828929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.828933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.828944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.828948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.828951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.828958:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.828964:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.828979:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760724.829056:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 22642688<>22646784 00000100:00100000:3.0:1789760724.829117:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688704:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.887124:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972847 gen 3 00000008:00010000:3.0:1789760724.887166:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760724.887173:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.887183:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.887278:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760724.887306:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688704:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.889518:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5528, 5528] 00000080:00200000:2.0:1789760724.889536:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.889541:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.889555:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22646784 i_blocks 45063 00000020:00200000:2.0:1789760724.889560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.889572:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.889599:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22646784, count: 4096 00000080:00200000:2.0:1789760724.889605:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22646784, bytes: 4096 00000080:00200000:2.0:1789760724.889618:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.889625:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.889627:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22646784, 22650880) 0 0 00000080:00200000:2.0:1789760724.891499:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.891518:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5529, 5529] 00020000:00200000:2.0:1789760724.891527:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22646784 chunk: [22646784, 22650880) 22650880, 4096 00020000:00200000:2.0:1789760724.891531:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.891550:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5529, 5529] 00000008:00010000:2.0:1789760724.891567:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.891572:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.891581:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22646784, 22650880) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.891593:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22646784=0x1599000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.891651:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.891654:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22650880 00000020:00010000:2.0:1789760724.891661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.891665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.891668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.891671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.891674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.891683:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.891689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.891717:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760724.891805:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 22646784<>22650880 00000100:00100000:3.0:1789760724.891895:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688832:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.921259:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972848 gen 3 00000008:00010000:3.0:1789760724.921364:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760724.921372:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.921383:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.921502:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760724.921528:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493688832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.924865:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5529, 5529] 00000080:00200000:2.0:1789760724.924891:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.924896:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.924910:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22650880 i_blocks 45063 00000020:00200000:2.0:1789760724.924915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.924929:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.924957:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22650880, count: 4096 00000080:00200000:2.0:1789760724.924964:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22650880, bytes: 4096 00000080:00200000:2.0:1789760724.924977:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.924984:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.924987:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22650880, 22654976) 0 0 00000080:00200000:2.0:1789760724.924990:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.925003:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5530, 5530] 00020000:00200000:2.0:1789760724.925010:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22650880 chunk: [22650880, 22654976) 22654976, 4096 00020000:00200000:2.0:1789760724.925014:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.925029:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5530, 5530] 00000008:00010000:2.0:1789760724.925042:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.925047:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.925057:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22650880, 22654976) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.925067:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22650880=0x159a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.925121:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.925124:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22654976 00000020:00010000:2.0:1789760724.925130:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.925134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.925137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.925141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.925144:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.925151:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.925157:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.925184:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760724.925274:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 22650880<>22654976 00000100:00100000:3.0:1789760724.925425:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688960:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760724.975575:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972849 gen 3 00000008:00010000:3.0:1789760724.975609:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760724.975615:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760724.975624:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760724.975719:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760724.975745:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493688960:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760724.975972:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5530, 5530] 00000080:00200000:2.0:1789760724.975984:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760724.975989:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760724.976001:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22654976 i_blocks 45063 00000020:00200000:2.0:1789760724.976004:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760724.976013:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760724.976038:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22654976, count: 4096 00000080:00200000:2.0:1789760724.976044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22654976, bytes: 4096 00000080:00200000:2.0:1789760724.976052:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760724.976058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760724.976060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22654976, 22659072) 0 0 00000080:00200000:2.0:1789760724.976063:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760724.976075:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5531, 5531] 00020000:00200000:2.0:1789760724.976082:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22654976 chunk: [22654976, 22659072) 22659072, 4096 00020000:00200000:2.0:1789760724.976086:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760724.976100:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5531, 5531] 00000008:00010000:2.0:1789760724.976113:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760724.976118:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760724.976125:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22654976, 22659072) append 0 append lockpos 0 00000080:00200000:2.0:1789760724.976135:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22654976=0x159b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760724.976275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760724.976278:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22659072 00000020:00010000:2.0:1789760724.976285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760724.976288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.976291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760724.976294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760724.976296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760724.976302:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760724.976307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760724.976323:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760724.976434:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 22654976<>22659072 00000100:00100000:3.0:1789760724.978534:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689088:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.018239:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972850 gen 3 00000008:00010000:3.0:1789760725.018275:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760725.018281:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.018292:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.018312:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760725.018337:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689088:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.018425:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5531, 5531] 00000080:00200000:2.0:1789760725.018437:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.018442:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.018464:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22659072 i_blocks 45063 00000020:00200000:2.0:1789760725.018467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.018476:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.018502:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22659072, count: 4096 00000080:00200000:2.0:1789760725.018518:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22659072, bytes: 4096 00000080:00200000:2.0:1789760725.018526:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.018533:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.018535:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22659072, 22663168) 0 0 00000080:00200000:2.0:1789760725.018538:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.018550:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5532, 5532] 00020000:00200000:2.0:1789760725.018556:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22659072 chunk: [22659072, 22663168) 22663168, 4096 00020000:00200000:2.0:1789760725.018559:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.018573:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5532, 5532] 00000008:00010000:2.0:1789760725.018585:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.018592:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.018600:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22659072, 22663168) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.018610:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22659072=0x159c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.018653:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.018656:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22663168 00000020:00010000:2.0:1789760725.018662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.018666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.018668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.018672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.018675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.018682:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.018687:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.018701:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760725.018773:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22659072<>22663168 00000100:00100000:3.0:1789760725.018914:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760725.018926:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.055724:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972851 gen 3 00000008:00010000:3.0:1789760725.055761:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760725.055767:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.055776:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.055794:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760725.055830:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689216:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.055923:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5532, 5532] 00000080:00200000:2.0:1789760725.055993:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.055999:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.056011:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22663168 i_blocks 45063 00000020:00200000:2.0:1789760725.056015:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.056024:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.056053:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22663168, count: 4096 00000080:00200000:2.0:1789760725.056059:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22663168, bytes: 4096 00000080:00200000:2.0:1789760725.056068:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.056075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.056077:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22663168, 22667264) 0 0 00000080:00200000:2.0:1789760725.056082:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.056095:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5533, 5533] 00020000:00200000:2.0:1789760725.056102:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22663168 chunk: [22663168, 22667264) 22667264, 4096 00020000:00200000:2.0:1789760725.056106:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.056120:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5533, 5533] 00000008:00010000:2.0:1789760725.056134:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.056138:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.056146:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22663168, 22667264) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.056157:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22663168=0x159d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.056199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.056202:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22667264 00000020:00010000:2.0:1789760725.056208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.056212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.056214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.056218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.056221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.056226:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.056231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.056244:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760725.056313:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 22663168<>22667264 00000100:00100000:3.0:1789760725.056391:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760725.056434:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.100770:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972852 gen 3 00000008:00010000:3.0:1789760725.100805:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760725.100812:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.100829:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.100852:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760725.100878:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.101189:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5533, 5533] 00000080:00200000:2.0:1789760725.101204:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.101210:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.101223:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22667264 i_blocks 45063 00000020:00200000:2.0:1789760725.101226:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.101236:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.101260:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22667264, count: 4096 00000080:00200000:2.0:1789760725.101268:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22667264, bytes: 4096 00000080:00200000:2.0:1789760725.101276:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.101283:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.101286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22667264, 22671360) 0 0 00000080:00200000:2.0:1789760725.101290:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.101303:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5534, 5534] 00020000:00200000:2.0:1789760725.101309:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22667264 chunk: [22667264, 22671360) 22671360, 4096 00020000:00200000:2.0:1789760725.101313:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.101329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5534, 5534] 00000008:00010000:2.0:1789760725.101343:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.101347:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.101356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22667264, 22671360) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.101366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22667264=0x159e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.101434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.101437:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22671360 00000020:00010000:2.0:1789760725.101444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.101448:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.101450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.101465:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.101468:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.101475:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.101480:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.101682:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:3.0:1789760725.101754:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22667264<>22671360 00000100:00100000:3.0:1789760725.101803:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.148856:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972853 gen 3 00000008:00010000:3.0:1789760725.148894:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760725.148900:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.148910:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.148928:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760725.148952:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689472:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.149149:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5534, 5534] 00000080:00200000:2.0:1789760725.149162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.149168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.149189:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22671360 i_blocks 45063 00000020:00200000:2.0:1789760725.149194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.149203:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.149230:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22671360, count: 4096 00000080:00200000:2.0:1789760725.149236:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22671360, bytes: 4096 00000080:00200000:2.0:1789760725.149245:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.149252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.149254:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22671360, 22675456) 0 0 00000080:00200000:2.0:1789760725.149258:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.149271:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5535, 5535] 00020000:00200000:2.0:1789760725.149277:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22671360 chunk: [22671360, 22675456) 22675456, 4096 00020000:00200000:2.0:1789760725.149281:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.149295:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5535, 5535] 00000008:00010000:2.0:1789760725.149309:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.149313:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.149322:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22671360, 22675456) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.149332:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22671360=0x159f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.149374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.149377:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22675456 00000020:00010000:2.0:1789760725.149384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.149387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.149390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.149437:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.149440:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.149446:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.149461:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.149484:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760725.149565:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 22671360<>22675456 00000100:00100000:3.0:1789760725.149759:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.218549:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972854 gen 3 00000008:00010000:3.0:1789760725.218584:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760725.218590:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.218600:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.218617:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760725.218644:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.218741:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5535, 5535] 00000080:00200000:2.0:1789760725.218756:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.218761:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.218773:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22675456 i_blocks 45063 00000020:00200000:2.0:1789760725.218777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.218786:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.218824:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22675456, count: 4096 00000080:00200000:2.0:1789760725.218830:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22675456, bytes: 4096 00000080:00200000:2.0:1789760725.218838:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.218846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.218848:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22675456, 22679552) 0 0 00000080:00200000:2.0:1789760725.218852:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.218864:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5536, 5536] 00020000:00200000:2.0:1789760725.218871:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22675456 chunk: [22675456, 22679552) 22679552, 4096 00020000:00200000:2.0:1789760725.218875:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.218890:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5536, 5536] 00000008:00010000:2.0:1789760725.218902:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.218906:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.218915:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22675456, 22679552) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.218925:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22675456=0x15a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.219070:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.219073:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22679552 00000020:00010000:2.0:1789760725.219081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.219084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.219088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.219091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.219094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.219099:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.219105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.219117:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760725.219190:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22675456<>22679552 00000100:00100000:3.0:1789760725.219248:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689728:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.267984:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972855 gen 3 00000008:00010000:3.0:1789760725.268023:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760725.268029:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.268039:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.268102:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760725.268130:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689728:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.268217:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5536, 5536] 00000080:00200000:2.0:1789760725.268231:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.268236:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.268247:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22679552 i_blocks 45063 00000020:00200000:2.0:1789760725.268250:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.268260:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.268286:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22679552, count: 4096 00000080:00200000:2.0:1789760725.268292:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22679552, bytes: 4096 00000080:00200000:2.0:1789760725.268301:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.268308:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.268311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22679552, 22683648) 0 0 00000080:00200000:2.0:1789760725.268315:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.268328:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5537, 5537] 00020000:00200000:2.0:1789760725.268335:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22679552 chunk: [22679552, 22683648) 22683648, 4096 00020000:00200000:2.0:1789760725.268339:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.268353:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5537, 5537] 00000008:00010000:2.0:1789760725.268367:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.268372:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.268380:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22679552, 22683648) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.268390:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22679552=0x15a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.268468:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.268471:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22683648 00000020:00010000:2.0:1789760725.268477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.268480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.268483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.268488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.268491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.268497:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.268503:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.268578:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760725.268656:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 22679552<>22683648 00000100:00100000:3.0:1789760725.268722:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.322790:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972856 gen 3 00000008:00010000:3.0:1789760725.322837:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760725.322844:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.322854:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.322871:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760725.322895:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493689856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.323043:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5537, 5537] 00000080:00200000:2.0:1789760725.323056:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.323061:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.323073:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22683648 i_blocks 45063 00000020:00200000:2.0:1789760725.323077:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.323086:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.323111:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22683648, count: 4096 00000080:00200000:2.0:1789760725.323117:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22683648, bytes: 4096 00000080:00200000:2.0:1789760725.323125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.323132:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.323134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22683648, 22687744) 0 0 00000080:00200000:2.0:1789760725.323138:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.323150:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5538, 5538] 00020000:00200000:2.0:1789760725.323157:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22683648 chunk: [22683648, 22687744) 22687744, 4096 00020000:00200000:2.0:1789760725.323161:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.323175:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5538, 5538] 00000008:00010000:2.0:1789760725.323188:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.323193:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.323202:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22683648, 22687744) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.323212:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22683648=0x15a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.323254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.323257:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22687744 00000020:00010000:2.0:1789760725.323264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.323268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.323271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.323275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.323277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.323285:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.323290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.323303:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760725.323376:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 22683648<>22687744 00000100:00100000:3.0:1789760725.323551:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.371640:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972857 gen 3 00000008:00010000:3.0:1789760725.371676:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760725.371683:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.371694:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.371713:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760725.371737:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493689984:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.371830:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5538, 5538] 00000080:00200000:2.0:1789760725.371841:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.371846:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.371857:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22687744 i_blocks 45063 00000020:00200000:2.0:1789760725.371860:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.371869:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.371899:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22687744, count: 4096 00000080:00200000:2.0:1789760725.371904:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22687744, bytes: 4096 00000080:00200000:2.0:1789760725.371913:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.371921:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.371923:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22687744, 22691840) 0 0 00000080:00200000:2.0:1789760725.371927:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.371952:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5539, 5539] 00020000:00200000:2.0:1789760725.371958:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22687744 chunk: [22687744, 22691840) 22691840, 4096 00020000:00200000:2.0:1789760725.371962:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.371977:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5539, 5539] 00000008:00010000:2.0:1789760725.371991:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.371996:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.372004:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22687744, 22691840) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.372015:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22687744=0x15a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.372072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.372075:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22691840 00000020:00010000:2.0:1789760725.372082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.372085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.372087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.372091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.372093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.372100:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.372105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.372119:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760725.372192:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 22687744<>22691840 00000100:00100000:3.0:1789760725.372448:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690112:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760725.453808:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760725.453820:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.453831:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.453857:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760725.453882:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690112:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.454089:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5539, 5539] 00000080:00200000:2.0:1789760725.454105:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.454110:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.454123:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22691840 i_blocks 45063 00000020:00200000:2.0:1789760725.454126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.454135:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.454163:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22691840, count: 4096 00000080:00200000:2.0:1789760725.454170:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22691840, bytes: 4096 00000080:00200000:2.0:1789760725.454179:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.454187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.454189:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22691840, 22695936) 0 0 00000080:00200000:2.0:1789760725.454193:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.454207:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5540, 5540] 00020000:00200000:2.0:1789760725.454216:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22691840 chunk: [22691840, 22695936) 22695936, 4096 00020000:00200000:2.0:1789760725.454221:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.454238:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5540, 5540] 00000008:00010000:2.0:1789760725.454253:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.454258:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.454269:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22691840, 22695936) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.454280:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22691840=0x15a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.454326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.454328:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22695936 00000020:00010000:2.0:1789760725.454335:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.454338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.454341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.454344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.454347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.454353:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.454358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.454371:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760725.454884:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 22691840<>22695936 00000100:00100000:3.0:1789760725.455694:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.537148:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972859 gen 3 00000008:00010000:3.0:1789760725.537192:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760725.537200:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.537208:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.537226:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760725.537253:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690240:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.537415:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5540, 5540] 00000080:00200000:2.0:1789760725.537428:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.537434:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.537445:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22695936 i_blocks 45063 00000020:00200000:2.0:1789760725.537457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.537467:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.537493:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22695936, count: 4096 00000080:00200000:2.0:1789760725.537500:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22695936, bytes: 4096 00000080:00200000:2.0:1789760725.537508:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.537516:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.537518:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22695936, 22700032) 0 0 00000080:00200000:2.0:1789760725.537522:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.537536:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5541, 5541] 00020000:00200000:2.0:1789760725.537543:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22695936 chunk: [22695936, 22700032) 22700032, 4096 00020000:00200000:2.0:1789760725.537547:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.537564:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5541, 5541] 00000008:00010000:2.0:1789760725.537580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.537585:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.537594:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22695936, 22700032) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.537606:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22695936=0x15a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.537652:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.537655:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22700032 00000020:00010000:2.0:1789760725.537663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.537665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.537668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.537671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.537674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.537679:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.537684:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.537698:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760725.537774:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 22695936<>22700032 00000100:00100000:3.0:1789760725.537885:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690368:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.607622:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972860 gen 3 00000008:00010000:3.0:1789760725.607675:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760725.607682:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.607693:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.607714:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760725.607744:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690368:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.607942:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5541, 5541] 00000080:00200000:2.0:1789760725.607955:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.607961:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.607972:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22700032 i_blocks 45063 00000020:00200000:2.0:1789760725.607976:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.607984:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.608011:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22700032, count: 4096 00000080:00200000:2.0:1789760725.608018:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22700032, bytes: 4096 00000080:00200000:2.0:1789760725.608028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.608035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.608037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22700032, 22704128) 0 0 00000080:00200000:2.0:1789760725.608041:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.608053:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5542, 5542] 00020000:00200000:2.0:1789760725.608059:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22700032 chunk: [22700032, 22704128) 22704128, 4096 00020000:00200000:2.0:1789760725.608063:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.608077:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5542, 5542] 00000008:00010000:2.0:1789760725.608091:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.608095:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.608105:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22700032, 22704128) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.608116:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22700032=0x15a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.608159:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.608161:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22704128 00000020:00010000:2.0:1789760725.608168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.608171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.608174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.608177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.608180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.608186:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.608192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.608205:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760725.608277:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 22700032<>22704128 00000100:00100000:2.0:1789760725.608315:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690496:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.642074:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972861 gen 3 00000008:00010000:3.0:1789760725.642123:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760725.642131:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.642143:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.642162:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760725.642190:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690496:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.642533:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5542, 5542] 00000080:00200000:2.0:1789760725.642552:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.642558:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.642574:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22704128 i_blocks 45063 00000020:00200000:2.0:1789760725.642578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.642589:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.642623:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22704128, count: 4096 00000080:00200000:2.0:1789760725.642631:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22704128, bytes: 4096 00000080:00200000:2.0:1789760725.642641:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.642649:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.642652:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22704128, 22708224) 0 0 00000080:00200000:2.0:1789760725.642657:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.642676:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5543, 5543] 00020000:00200000:2.0:1789760725.642685:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22704128 chunk: [22704128, 22708224) 22708224, 4096 00020000:00200000:2.0:1789760725.642690:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.642712:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5543, 5543] 00000008:00010000:2.0:1789760725.642730:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.642736:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.642747:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22704128, 22708224) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.642774:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22704128=0x15a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.642847:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.642850:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22708224 00000020:00010000:2.0:1789760725.642859:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.642864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.642867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.642871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.642875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.642883:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.642891:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.642908:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760725.643021:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 22704128<>22708224 00000100:00100000:3.0:1789760725.643465:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690624:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760725.686349:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760725.686369:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.686382:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.686480:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760725.686576:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690624:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.686648:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5543, 5543] 00000080:00200000:2.0:1789760725.686666:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.686672:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.686690:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22708224 i_blocks 45063 00000020:00200000:2.0:1789760725.686695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.686706:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.686742:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22708224, count: 4096 00000080:00200000:2.0:1789760725.686749:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22708224, bytes: 4096 00000080:00200000:2.0:1789760725.686759:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.686769:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.686772:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22708224, 22712320) 0 0 00000080:00200000:2.0:1789760725.686777:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.686795:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5544, 5544] 00020000:00200000:2.0:1789760725.686805:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22708224 chunk: [22708224, 22712320) 22712320, 4096 00020000:00200000:2.0:1789760725.686809:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.686885:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5544, 5544] 00000008:00010000:2.0:1789760725.686905:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.686912:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.686925:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22708224, 22712320) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.686939:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22708224=0x15a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.687003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.687006:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22712320 00000020:00010000:2.0:1789760725.687015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.687020:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.687023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.687027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.687030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.687038:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.687045:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.687064:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760725.687182:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 22708224<>22712320 00000100:00100000:3.0:1789760725.687250:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690752:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.717093:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972862 gen 3 00000100:00100000:3.0:1789760725.717137:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83800 x1876699493690752/t4294972863(4294972863) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760741 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760725.717187:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760725.717194:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.717206:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.717226:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760725.717253:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493690752:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.717853:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5544, 5544] 00000080:00200000:2.0:1789760725.717869:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.717875:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.717891:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22712320 i_blocks 45063 00000020:00200000:2.0:1789760725.717894:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.717905:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.717947:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22712320, count: 4096 00000080:00200000:2.0:1789760725.717955:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22712320, bytes: 4096 00000080:00200000:2.0:1789760725.717965:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.717973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.717975:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22712320, 22716416) 0 0 00000080:00200000:2.0:1789760725.717980:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.717995:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5545, 5545] 00020000:00200000:2.0:1789760725.718003:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22712320 chunk: [22712320, 22716416) 22716416, 4096 00020000:00200000:2.0:1789760725.718007:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.718026:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5545, 5545] 00000008:00010000:2.0:1789760725.718045:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.718051:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.718063:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22712320, 22716416) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.718077:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22712320=0x15a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.718133:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.718136:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22716416 00000020:00010000:2.0:1789760725.718144:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.718148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.718152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.718156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.718159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.718167:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.718174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.718190:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760725.718293:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22712320<>22716416 00000100:00100000:3.0:1789760725.718436:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690880:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.746806:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972864 gen 3 00000008:00010000:3.0:1789760725.746892:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760725.746901:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.746913:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.746939:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760725.746970:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493690880:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.747177:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5545, 5545] 00000080:00200000:2.0:1789760725.747196:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.747203:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.747219:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22716416 i_blocks 45063 00000020:00200000:2.0:1789760725.747223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.747234:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.747271:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22716416, count: 4096 00000080:00200000:2.0:1789760725.747279:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22716416, bytes: 4096 00000080:00200000:2.0:1789760725.747290:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.747299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.747302:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22716416, 22720512) 0 0 00000080:00200000:2.0:1789760725.747307:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.747325:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5546, 5546] 00020000:00200000:2.0:1789760725.747335:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22716416 chunk: [22716416, 22720512) 22720512, 4096 00020000:00200000:2.0:1789760725.747338:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.747357:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5546, 5546] 00000008:00010000:2.0:1789760725.747374:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.747380:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.747421:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22716416, 22720512) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.747436:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22716416=0x15aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.747509:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.747512:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22720512 00000020:00010000:2.0:1789760725.747520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.747525:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.747527:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.747530:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.747533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.747540:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.747546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.747561:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760725.747671:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22716416<>22720512 00000100:00100000:3.0:1789760725.747864:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691008:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.776619:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972865 gen 3 00000008:00010000:3.0:1789760725.776674:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760725.776683:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.776694:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.776714:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760725.776741:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691008:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.776836:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5546, 5546] 00000080:00200000:2.0:1789760725.776852:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.776858:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.776872:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22720512 i_blocks 45063 00000020:00200000:2.0:1789760725.776875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.776886:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.776919:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22720512, count: 4096 00000080:00200000:2.0:1789760725.776935:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22720512, bytes: 4096 00000080:00200000:2.0:1789760725.776944:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.776954:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.776956:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22720512, 22724608) 0 0 00000080:00200000:2.0:1789760725.776960:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.776975:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5547, 5547] 00020000:00200000:2.0:1789760725.776984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22720512 chunk: [22720512, 22724608) 22724608, 4096 00020000:00200000:2.0:1789760725.776989:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.777007:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5547, 5547] 00000008:00010000:2.0:1789760725.777023:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.777030:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.777042:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22720512, 22724608) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.777056:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22720512=0x15ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.777115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.777118:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22724608 00000020:00010000:2.0:1789760725.777127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.777132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.777136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.777139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.777141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.777149:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.777155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.777169:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760725.777263:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22720512<>22724608 00000100:00100000:3.0:1789760725.777530:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691136:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.819318:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972866 gen 3 00000008:00010000:3.0:1789760725.819372:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760725.819380:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.819430:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.819458:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760725.819492:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691136:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.819573:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5547, 5547] 00000080:00200000:2.0:1789760725.819590:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.819597:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.819615:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22724608 i_blocks 45063 00000020:00200000:2.0:1789760725.819630:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.819642:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.819677:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22724608, count: 4096 00000080:00200000:2.0:1789760725.819685:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22724608, bytes: 4096 00000080:00200000:2.0:1789760725.819695:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.819703:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.819706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22724608, 22728704) 0 0 00000080:00200000:2.0:1789760725.819710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.819726:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5548, 5548] 00020000:00200000:2.0:1789760725.819735:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22724608 chunk: [22724608, 22728704) 22728704, 4096 00020000:00200000:2.0:1789760725.819740:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.819759:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5548, 5548] 00000008:00010000:2.0:1789760725.819780:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.819786:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.819797:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22724608, 22728704) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.819812:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22724608=0x15ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.819870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.819873:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22728704 00000020:00010000:2.0:1789760725.819880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.819884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.819887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.819891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.819894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.819902:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.819909:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.819926:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760725.820043:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 22724608<>22728704 00000100:00100000:3.0:1789760725.820124:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691264:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760725.861713:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760725.861731:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.861745:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.861929:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760725.861967:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691264:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760725.863529:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493691392:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760725.899877:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972868 gen 3 00000008:00010000:1.0:1789760725.899958:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760725.899968:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760725.899979:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760725.900002:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000080:00200000:3.0:1789760725.900029:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5549, 5549] 00000100:00100000:1.0:1789760725.900032:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493691392:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760725.900047:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760725.900053:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760725.900068:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22732800 i_blocks 45063 00000020:00200000:3.0:1789760725.900071:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760725.900083:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760725.900118:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22732800, count: 4096 00000080:00200000:3.0:1789760725.900126:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22732800, bytes: 4096 00000080:00200000:3.0:1789760725.900135:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760725.900144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760725.900146:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22732800, 22736896) 0 0 00000080:00200000:3.0:1789760725.900152:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760725.900169:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5550, 5550] 00020000:00200000:3.0:1789760725.900178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22732800 chunk: [22732800, 22736896) 22736896, 4096 00020000:00200000:3.0:1789760725.900182:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760725.900201:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5550, 5550] 00000008:00010000:3.0:1789760725.900219:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760725.900224:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760725.900235:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22732800, 22736896) append 0 append lockpos 0 00000080:00200000:3.0:1789760725.900248:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22732800=0x15ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760725.900307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760725.900311:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22736896 00000020:00010000:3.0:1789760725.900319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:3.0:1789760725.900323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.900326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.900330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760725.900333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760725.900340:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760725.900346:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.900364:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760725.900685:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22732800<>22736896 00000100:00100000:3.0:1789760725.900742:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691520:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760725.951189:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972869 gen 3 00000008:00010000:3.0:1789760725.951243:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760725.951252:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760725.951264:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760725.951295:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760725.951325:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691520:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760725.951595:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5550, 5550] 00000080:00200000:2.0:1789760725.951613:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.951619:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.951635:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22736896 i_blocks 45063 00000020:00200000:2.0:1789760725.951639:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.951649:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.951685:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22736896, count: 4096 00000080:00200000:2.0:1789760725.951692:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22736896, bytes: 4096 00000080:00200000:2.0:1789760725.951702:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.951711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.951713:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22736896, 22740992) 0 0 00000080:00200000:2.0:1789760725.951718:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.951733:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5551, 5551] 00020000:00200000:2.0:1789760725.951741:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22736896 chunk: [22736896, 22740992) 22740992, 4096 00020000:00200000:2.0:1789760725.951745:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.951764:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5551, 5551] 00000008:00010000:2.0:1789760725.951782:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.951789:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760725.951801:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22736896, 22740992) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.951815:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22736896=0x15af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.951874:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.951877:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22740992 00000020:00010000:2.0:1789760725.951885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.951889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.951893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b45a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.951897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac6c8 (ffff8eb6450b4008/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.951900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.951908:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.951915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.951930:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760725.952111:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22736896<>22740992 00000100:00100000:3.0:1789760725.952439:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691648:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.022905:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972870 gen 3 00000008:00010000:2.0:1789760726.023131:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:2.0:1789760726.023140:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.023153:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.023178:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:2.0:1789760726.023209:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493691648:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760726.023218:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5551, 5551] 00000080:00200000:3.0:1789760726.023234:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.023240:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.023255:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22740992 i_blocks 45063 00000020:00200000:3.0:1789760726.023259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.023322:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.023434:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22740992, count: 4096 00000080:00200000:2.0:1789760726.023442:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22740992, bytes: 4096 00000080:00200000:2.0:1789760726.023464:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.023473:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.023476:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22740992, 22745088) 0 0 00000080:00200000:2.0:1789760726.023481:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.023499:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5552, 5552] 00020000:00200000:2.0:1789760726.023508:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22740992 chunk: [22740992, 22745088) 22745088, 4096 00020000:00200000:2.0:1789760726.023512:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.023645:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5552, 5552] 00000008:00010000:2.0:1789760726.023668:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.023674:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.023686:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22740992, 22745088) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.023702:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22740992=0x15b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.023756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.023759:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22745088 00000020:00010000:2.0:1789760726.023767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd4e8 (ffff8eb650f97128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.023771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd4e8 (ffff8eb650f97128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.023775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.023779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd4e8 (ffff8eb650f97128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.023782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.023791:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.023798:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.023812:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760726.023916:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22740992<>22745088 00000100:00100000:3.0:1789760726.024388:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691776:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760726.074234:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:2.0:1789760726.074250:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.074265:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.074290:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:2.0:1789760726.074324:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493691776:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.075140:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760726.075161:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493691904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.110143:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972871 gen 3 00000100:00100000:1.0:1789760726.110311:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304e680 x1876699493691904/t4294972872(4294972872) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760742 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760726.110366:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760726.110374:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.110384:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.110507:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760726.110532:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493691904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.113059:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.170324:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972873 gen 3 00000008:00010000:1.0:1789760726.170424:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760726.170438:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.170457:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.170480:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760726.170519:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.171322:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493692160:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760726.238048:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760726.238065:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.238078:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.238161:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760726.238193:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493692160:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760726.244027:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5555, 5555] 00000080:00200000:3.0:1789760726.244054:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.244062:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.244081:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22757376 i_blocks 45063 00000020:00200000:3.0:1789760726.244087:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.244106:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.244170:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22757376, count: 4096 00000080:00200000:3.0:1789760726.244179:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22757376, bytes: 4096 00000080:00200000:3.0:1789760726.244192:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.244201:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.244203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22757376, 22761472) 0 0 00000080:00200000:3.0:1789760726.244208:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.244225:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5556, 5556] 00020000:00200000:3.0:1789760726.244234:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22757376 chunk: [22757376, 22761472) 22761472, 4096 00020000:00200000:3.0:1789760726.244238:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.244260:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5556, 5556] 00000008:00010000:3.0:1789760726.244279:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.244286:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760726.244297:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22757376, 22761472) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.244314:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22757376=0x15b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.244384:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.244387:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22761472 00000020:00010000:3.0:1789760726.244659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.244664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.244668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.244672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.244675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.244684:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.244691:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.244719:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760726.244909:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 22757376<>22761472 00000100:00100000:2.0:1789760726.245053:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493692288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.295730:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972875 gen 3 00000008:00010000:3.0:1789760726.295796:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760726.295805:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.295825:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.295843:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760726.295872:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493692288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.296131:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5556, 5556] 00000080:00200000:2.0:1789760726.296151:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.296156:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.296173:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22761472 i_blocks 45063 00000020:00200000:2.0:1789760726.296176:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.296186:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.296222:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22761472, count: 4096 00000080:00200000:2.0:1789760726.296229:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22761472, bytes: 4096 00000080:00200000:2.0:1789760726.296239:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.296248:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.296251:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22761472, 22765568) 0 0 00000080:00200000:2.0:1789760726.296255:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.296273:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5557, 5557] 00020000:00200000:2.0:1789760726.296282:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22761472 chunk: [22761472, 22765568) 22765568, 4096 00020000:00200000:2.0:1789760726.296287:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.296307:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5557, 5557] 00000008:00010000:2.0:1789760726.296325:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.296331:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.296344:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22761472, 22765568) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.296359:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22761472=0x15b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.296459:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.296462:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22765568 00000020:00010000:2.0:1789760726.296470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.296474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.296477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.296481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.296484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.296491:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.296499:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.296514:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760726.296687:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 22761472<>22765568 00000100:00100000:3.0:1789760726.297111:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493692416:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760726.361629:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760726.361647:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.361662:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.361685:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760726.361721:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493692416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.361808:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5557, 5557] 00000080:00200000:2.0:1789760726.361836:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.361843:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.361858:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22765568 i_blocks 45063 00000020:00200000:2.0:1789760726.361862:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.361871:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.361901:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22765568, count: 4096 00000080:00200000:2.0:1789760726.361909:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22765568, bytes: 4096 00000080:00200000:2.0:1789760726.361919:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.361928:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.361930:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22765568, 22769664) 0 0 00000080:00200000:2.0:1789760726.361935:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.362165:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5558, 5558] 00020000:00200000:2.0:1789760726.362174:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22765568 chunk: [22765568, 22769664) 22769664, 4096 00020000:00200000:2.0:1789760726.362178:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.362200:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5558, 5558] 00000008:00010000:2.0:1789760726.362220:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.362227:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.362241:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22765568, 22769664) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.362255:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22765568=0x15b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.362315:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.362318:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22769664 00000020:00010000:2.0:1789760726.362327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.362331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.362334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.362338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.362341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.362349:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.362357:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.362373:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760726.362741:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 22765568<>22769664 00000100:00100000:2.0:1789760726.362789:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493692544:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.399898:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972877 gen 3 00000008:00010000:3.0:1789760726.400218:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760726.400228:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.400240:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.400263:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760726.400293:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493692544:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.402498:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.452196:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972878 gen 3 00000008:00010000:1.0:1789760726.452252:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760726.452262:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.452274:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.452302:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760726.452332:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.453294:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760726.453311:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692800:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.494686:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972879 gen 3 00000008:00010000:1.0:1789760726.494738:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760726.494747:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.494758:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.494780:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760726.494809:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692800:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.496250:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.537086:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972880 gen 3 00000008:00010000:1.0:1789760726.537139:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760726.537149:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.537161:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.537242:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760726.537273:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493692928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.541285:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493693056:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.572775:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493693312:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760726.573288:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493693184:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760726.573341:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493693440:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760726.576518:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493693184:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760726.576558:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493693440:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760726.585804:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493693312:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760726.586121:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972881 gen 3 00000008:00010000:1.0:1789760726.586161:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760726.586170:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.586181:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.586243:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760726.586270:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493693056:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.586341:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5562, 5562] 00000080:00200000:2.0:1789760726.586358:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.586364:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.586378:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22786048 i_blocks 45063 00000020:00200000:2.0:1789760726.586382:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.586412:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.586456:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22786048, count: 4096 00000080:00200000:2.0:1789760726.586464:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22786048, bytes: 4096 00000080:00200000:2.0:1789760726.586475:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.586484:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.586487:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22786048, 22790144) 0 0 00000080:00200000:2.0:1789760726.586491:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.586581:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5563, 5563] 00020000:00200000:2.0:1789760726.586590:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22786048 chunk: [22786048, 22790144) 22790144, 4096 00020000:00200000:2.0:1789760726.586595:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.586616:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5563, 5563] 00000008:00010000:2.0:1789760726.586636:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.586643:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.586655:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22786048, 22790144) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.586670:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22786048=0x15bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.586729:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.586731:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22790144 00000020:00010000:2.0:1789760726.586741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.586745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.586748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.586751:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.586754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.586762:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.586770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.586785:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760726.586891:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 22786048<>22790144 00000100:00100000:3.0:1789760726.586965:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493693568:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.629994:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972882 gen 3 00000008:00010000:3.0:1789760726.630045:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760726.630053:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.630063:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.630123:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760726.630153:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493693568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.630236:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5563, 5563] 00000080:00200000:2.0:1789760726.630253:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.630259:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.630276:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22790144 i_blocks 45063 00000020:00200000:2.0:1789760726.630280:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.630293:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.630329:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22790144, count: 4096 00000080:00200000:2.0:1789760726.630337:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22790144, bytes: 4096 00000080:00200000:2.0:1789760726.630346:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.630355:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.630357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22790144, 22794240) 0 0 00000080:00200000:2.0:1789760726.630363:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.630379:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5564, 5564] 00020000:00200000:2.0:1789760726.630387:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22790144 chunk: [22790144, 22794240) 22794240, 4096 00020000:00200000:2.0:1789760726.630392:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.630608:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5564, 5564] 00000008:00010000:2.0:1789760726.630625:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.630631:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.630643:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22790144, 22794240) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.630656:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22790144=0x15bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.630713:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.630716:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22794240 00000020:00010000:2.0:1789760726.630724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.630728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.630732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.630735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.630737:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.630745:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.630751:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.630767:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760726.630933:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22790144<>22794240 00000100:00100000:3.0:1789760726.631000:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.665490:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972883 gen 3 00000008:00010000:2.0:1789760726.665697:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760726.665707:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.665717:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.665739:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760726.665770:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693696:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760726.665797:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5564, 5564] 00000080:00200000:3.0:1789760726.665817:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.665824:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.665845:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22794240 i_blocks 45063 00000020:00200000:3.0:1789760726.665850:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.665938:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.665976:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22794240, count: 4096 00000080:00200000:2.0:1789760726.665985:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22794240, bytes: 4096 00000080:00200000:2.0:1789760726.665995:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.666006:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.666008:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22794240, 22798336) 0 0 00000080:00200000:2.0:1789760726.666013:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.666030:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5565, 5565] 00020000:00200000:2.0:1789760726.666039:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22794240 chunk: [22794240, 22798336) 22798336, 4096 00020000:00200000:2.0:1789760726.666044:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.666063:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5565, 5565] 00000008:00010000:2.0:1789760726.666093:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.666100:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.666113:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22794240, 22798336) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.666128:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22794240=0x15bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.666185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.666188:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22798336 00000020:00010000:2.0:1789760726.666197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.666200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.666204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.666208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.666211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.666219:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.666228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.666522:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760726.666627:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 22794240<>22798336 00000100:00100000:2.0:1789760726.666671:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760726.666689:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693824:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760726.726035:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760726.726051:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.726065:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.726123:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760726.726157:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.726311:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5565, 5565] 00000080:00200000:2.0:1789760726.726327:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.726335:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.726813:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22798336 i_blocks 45063 00000020:00200000:3.0:1789760726.726820:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.726832:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.726865:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22798336, count: 4096 00000080:00200000:3.0:1789760726.726875:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22798336, bytes: 4096 00000080:00200000:3.0:1789760726.726886:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.726894:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.726897:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22798336, 22802432) 0 0 00000080:00200000:3.0:1789760726.726902:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.726918:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5566, 5566] 00020000:00200000:3.0:1789760726.726926:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22798336 chunk: [22798336, 22802432) 22802432, 4096 00020000:00200000:3.0:1789760726.726931:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.726953:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5566, 5566] 00000008:00010000:3.0:1789760726.726970:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.726976:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760726.726989:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22798336, 22802432) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.727004:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22798336=0x15be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.727063:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.727066:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22802432 00000020:00010000:3.0:1789760726.727074:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.727078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.727081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.727085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.727089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.727098:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.727105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.727119:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760726.727300:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 22798336<>22802432 00000100:00100000:2.0:1789760726.727367:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.767864:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972884 gen 3 00000100:00100000:3.0:1789760726.767900:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149180 x1876699493693952/t4294972885(4294972885) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760742 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760726.767961:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760726.767969:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.767978:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.767996:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760726.768019:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493693952:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.768332:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5566, 5566] 00000080:00200000:2.0:1789760726.768351:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.768358:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.768374:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22802432 i_blocks 45063 00000020:00200000:2.0:1789760726.768378:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.768389:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:1.0:1789760726.769484:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694080:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.826023:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972886 gen 3 00000008:00010000:1.0:1789760726.826091:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760726.826100:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.826120:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.826138:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760726.826167:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.826257:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5567, 5567] 00000080:00200000:2.0:1789760726.826274:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.826281:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.826297:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22806528 i_blocks 45063 00000020:00200000:2.0:1789760726.826301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.826310:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.826344:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22806528, count: 4096 00000080:00200000:2.0:1789760726.826351:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22806528, bytes: 4096 00000080:00200000:2.0:1789760726.826360:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.826368:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.826371:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22806528, 22810624) 0 0 00000080:00200000:2.0:1789760726.826376:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.826416:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5568, 5568] 00020000:00200000:2.0:1789760726.826425:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22806528 chunk: [22806528, 22810624) 22810624, 4096 00020000:00200000:2.0:1789760726.826430:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.826459:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5568, 5568] 00000008:00010000:2.0:1789760726.826696:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.826703:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.826717:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22806528, 22810624) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.826734:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22806528=0x15c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.826807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.826810:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22810624 00000020:00010000:2.0:1789760726.826820:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.826823:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.826826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.826830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.826845:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.826854:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.826861:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.826879:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760726.826991:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 22806528<>22810624 00000100:00100000:3.0:1789760726.827253:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493694208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.881347:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972887 gen 3 00000008:00010000:3.0:1789760726.882102:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760726.882113:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.882127:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.882148:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760726.882179:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493694208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760726.882292:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5568, 5568] 00000080:00200000:2.0:1789760726.882310:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.882316:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.882333:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22810624 i_blocks 45063 00000020:00200000:2.0:1789760726.882338:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.882349:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.882383:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22810624, count: 4096 00000080:00200000:2.0:1789760726.882391:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22810624, bytes: 4096 00000080:00200000:2.0:1789760726.882422:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.882431:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.882433:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22810624, 22814720) 0 0 00000080:00200000:2.0:1789760726.882438:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.882474:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5569, 5569] 00020000:00200000:2.0:1789760726.882483:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22810624 chunk: [22810624, 22814720) 22814720, 4096 00020000:00200000:2.0:1789760726.882487:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.883107:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5569, 5569] 00000008:00010000:2.0:1789760726.883129:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.883135:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760726.883148:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22810624, 22814720) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.883163:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22810624=0x15c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.883227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.883230:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22814720 00000020:00010000:2.0:1789760726.883239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.883243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.883246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.883249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.883252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.883262:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.883269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.883289:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760726.883909:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 22810624<>22814720 00000100:00100000:2.0:1789760726.883959:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760726.883983:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493694336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760726.946230:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972888 gen 3 00000008:00010000:3.0:1789760726.946283:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760726.946292:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760726.946304:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760726.946332:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760726.946363:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493694336:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.947628:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493694464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.987570:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972889 gen 3 00000008:00010000:1.0:1789760726.987625:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760726.987634:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.987646:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.987668:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760726.987704:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493694464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760726.988813:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.021619:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972890 gen 3 00000008:00010000:1.0:1789760727.021671:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760727.021681:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.021693:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.021717:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760727.021747:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.022760:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760727.022780:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.070356:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972891 gen 3 00000008:00010000:1.0:1789760727.070456:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760727.070466:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.070479:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.070555:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760727.070588:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.071699:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694848:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760727.102539:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760727.102556:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.102568:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.102645:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760727.102678:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493694848:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.103640:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493694976:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.135297:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972893 gen 3 00000008:00010000:1.0:1789760727.135365:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760727.135374:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.135384:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.135457:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760727.135487:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493694976:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760727.135646:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5574, 5574] 00000080:00200000:3.0:1789760727.135664:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.135669:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.135685:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22835200 i_blocks 45063 00000020:00200000:3.0:1789760727.135689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.135699:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.135734:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22835200, count: 4096 00000080:00200000:3.0:1789760727.135742:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22835200, bytes: 4096 00000080:00200000:3.0:1789760727.135751:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.135760:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.135762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22835200, 22839296) 0 0 00000080:00200000:3.0:1789760727.135767:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.135784:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5575, 5575] 00020000:00200000:3.0:1789760727.135792:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22835200 chunk: [22835200, 22839296) 22839296, 4096 00020000:00200000:3.0:1789760727.135797:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.135817:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5575, 5575] 00000008:00010000:3.0:1789760727.135835:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.135842:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760727.135854:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22835200, 22839296) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.135866:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22835200=0x15c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.135999:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.136002:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22839296 00000020:00010000:3.0:1789760727.136012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.136015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.136018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.136022:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.136025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.136033:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.136040:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.136058:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760727.136167:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 22835200<>22839296 00000100:00100000:3.0:1789760727.136218:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760727.136236:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493695104:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760727.179446:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760727.179464:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.179475:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.179502:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760727.179541:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493695104:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760727.179737:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5575, 5575] 00000080:00200000:1.0:1789760727.179752:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.179757:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.179779:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22839296 i_blocks 45063 00000020:00200000:1.0:1789760727.179783:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.179792:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.179824:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22839296, count: 4096 00000080:00200000:1.0:1789760727.179832:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22839296, bytes: 4096 00000080:00200000:1.0:1789760727.179928:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.179991:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.179994:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22839296, 22843392) 0 0 00000080:00200000:1.0:1789760727.179999:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.180015:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5576, 5576] 00020000:00200000:1.0:1789760727.180023:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22839296 chunk: [22839296, 22843392) 22843392, 4096 00020000:00200000:1.0:1789760727.180027:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.180044:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5576, 5576] 00000008:00010000:1.0:1789760727.180061:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.180067:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.180079:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22839296, 22843392) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.180103:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22839296=0x15c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.180154:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.180156:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22843392 00000020:00010000:1.0:1789760727.180164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.180167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.180170:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.180175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.180178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.180187:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.180192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.180207:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760727.180306:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 22839296<>22843392 00000100:00100000:1.0:1789760727.180351:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760727.180368:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493695232:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.229858:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972895 gen 3 00000008:00010000:1.0:1789760727.230043:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760727.230052:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.230063:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.230087:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760727.230118:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493695232:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760727.230266:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5576, 5576] 00000080:00200000:3.0:1789760727.230282:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.230291:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.230306:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22843392 i_blocks 45063 00000020:00200000:3.0:1789760727.230309:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.230319:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.230352:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22843392, count: 4096 00000080:00200000:3.0:1789760727.230360:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22843392, bytes: 4096 00000080:00200000:3.0:1789760727.230370:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.230380:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.230382:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22843392, 22847488) 0 0 00000080:00200000:3.0:1789760727.230387:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.230435:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5577, 5577] 00020000:00200000:3.0:1789760727.230452:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22843392 chunk: [22843392, 22847488) 22847488, 4096 00020000:00200000:3.0:1789760727.230457:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.230477:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5577, 5577] 00000008:00010000:3.0:1789760727.230495:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.230511:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760727.230523:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22843392, 22847488) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.230637:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22843392=0x15c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.230698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.230702:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22847488 00000020:00010000:3.0:1789760727.230710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.230713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.230716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.230720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.230723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.230730:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.230737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.230753:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760727.230904:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 22843392<>22847488 00000100:00100000:2.0:1789760727.231180:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493695360:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760727.282342:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760727.282732:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760727.282743:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.282755:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.282879:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760727.282910:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493695360:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760727.283033:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5577, 5577] 00000080:00200000:1.0:1789760727.283049:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.283056:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.283071:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22847488 i_blocks 45063 00000020:00200000:1.0:1789760727.283075:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.283084:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.283179:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22847488, count: 4096 00000080:00200000:1.0:1789760727.283187:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22847488, bytes: 4096 00000080:00200000:1.0:1789760727.283197:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.283206:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.283208:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22847488, 22851584) 0 0 00000080:00200000:1.0:1789760727.283212:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.283228:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5578, 5578] 00020000:00200000:1.0:1789760727.283237:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22847488 chunk: [22847488, 22851584) 22851584, 4096 00020000:00200000:1.0:1789760727.283241:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.283259:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5578, 5578] 00000008:00010000:1.0:1789760727.283319:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.283326:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.283338:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22847488, 22851584) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.283351:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22847488=0x15ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.283430:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.283433:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22851584 00000020:00010000:1.0:1789760727.283450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.283454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.283457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.283461:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.283464:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.283475:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.283483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.283499:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760727.283699:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 22847488<>22851584 00000080:00200000:1.0:1789760727.339624:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5578, 5578] 00000080:00200000:1.0:1789760727.339643:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.339649:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.339665:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22851584 i_blocks 45063 00000020:00200000:1.0:1789760727.339669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.339680:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.339715:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22851584, count: 4096 00000080:00200000:1.0:1789760727.339723:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22851584, bytes: 4096 00000080:00200000:1.0:1789760727.339735:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.339744:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.339747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22851584, 22855680) 0 0 00000080:00200000:1.0:1789760727.339751:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.339767:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5579, 5579] 00020000:00200000:1.0:1789760727.339776:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22851584 chunk: [22851584, 22855680) 22855680, 4096 00020000:00200000:1.0:1789760727.339781:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.339800:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5579, 5579] 00000008:00010000:1.0:1789760727.339819:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.339825:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.339837:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22851584, 22855680) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.339852:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22851584=0x15cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.339908:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.339910:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22855680 00000020:00010000:1.0:1789760727.346847:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.346863:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.346869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.346874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.346878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.346899:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.346914:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.347868:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760727.349542:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 22851584<>22855680 00000080:00200000:1.0:1789760727.387144:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5579, 5579] 00000080:00200000:1.0:1789760727.387165:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.387172:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.387190:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22855680 i_blocks 45063 00000020:00200000:1.0:1789760727.387195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.387205:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.387238:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22855680, count: 4096 00000080:00200000:1.0:1789760727.387246:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22855680, bytes: 4096 00000080:00200000:1.0:1789760727.387260:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.387269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.387271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22855680, 22859776) 0 0 00000080:00200000:1.0:1789760727.387276:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.387295:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5580, 5580] 00020000:00200000:1.0:1789760727.387303:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22855680 chunk: [22855680, 22859776) 22859776, 4096 00020000:00200000:1.0:1789760727.387308:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.387328:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5580, 5580] 00000008:00010000:1.0:1789760727.387350:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.387356:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.387369:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22855680, 22859776) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.387383:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22855680=0x15cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.387493:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.387496:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22859776 00000020:00010000:1.0:1789760727.387505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.387508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.387512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.387516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.387524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.387536:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.387543:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.387560:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760727.387664:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 22855680<>22859776 00000080:00200000:1.0:1789760727.436447:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5580, 5580] 00000080:00200000:1.0:1789760727.436463:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.436469:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.436486:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22859776 i_blocks 45063 00000020:00200000:1.0:1789760727.436499:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.436510:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.436551:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22859776, count: 4096 00000080:00200000:1.0:1789760727.436558:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22859776, bytes: 4096 00000080:00200000:1.0:1789760727.436567:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.436576:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.436578:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22859776, 22863872) 0 0 00000080:00200000:1.0:1789760727.436583:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.436598:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5581, 5581] 00020000:00200000:1.0:1789760727.436607:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22859776 chunk: [22859776, 22863872) 22863872, 4096 00020000:00200000:1.0:1789760727.436612:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.436631:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5581, 5581] 00000008:00010000:1.0:1789760727.436649:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.436654:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.436664:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22859776, 22863872) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.436677:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22859776=0x15cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.436733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.436736:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22863872 00000020:00010000:1.0:1789760727.436744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.436747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.436750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.436754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.436757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.436923:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.436933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.436958:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760727.437061:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 22859776<>22863872 00000080:00200000:1.0:1789760727.474368:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5581, 5581] 00000080:00200000:1.0:1789760727.474386:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.474433:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.474459:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22863872 i_blocks 45063 00000020:00200000:1.0:1789760727.474462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.474474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.474508:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22863872, count: 4096 00000080:00200000:1.0:1789760727.474515:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22863872, bytes: 4096 00000080:00200000:1.0:1789760727.474525:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.474534:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.474536:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22863872, 22867968) 0 0 00000080:00200000:1.0:1789760727.474659:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.474676:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5582, 5582] 00020000:00200000:1.0:1789760727.474685:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22863872 chunk: [22863872, 22867968) 22867968, 4096 00020000:00200000:1.0:1789760727.474689:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.474709:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5582, 5582] 00000008:00010000:1.0:1789760727.474726:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.474732:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.474744:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22863872, 22867968) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.474758:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22863872=0x15ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.474814:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.474817:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22867968 00000020:00010000:1.0:1789760727.474825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.474829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.474832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.474836:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.474839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.474850:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.474857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.474882:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760727.474998:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 22863872<>22867968 00000080:00200000:1.0:1789760727.508768:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5582, 5582] 00000080:00200000:1.0:1789760727.508784:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.508790:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.508805:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22867968 i_blocks 45063 00000020:00200000:1.0:1789760727.508809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.508881:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.509111:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22867968, count: 4096 00000080:00200000:1.0:1789760727.509120:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22867968, bytes: 4096 00000080:00200000:1.0:1789760727.509130:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.509140:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.509142:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22867968, 22872064) 0 0 00000080:00200000:1.0:1789760727.509146:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.509163:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5583, 5583] 00020000:00200000:1.0:1789760727.509172:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22867968 chunk: [22867968, 22872064) 22872064, 4096 00020000:00200000:1.0:1789760727.509176:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.509194:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5583, 5583] 00000008:00010000:1.0:1789760727.509215:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.509221:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.509232:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22867968, 22872064) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.509246:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22867968=0x15cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.509307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.509310:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22872064 00000020:00010000:1.0:1789760727.509319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.509322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.509326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.509329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.509331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.509342:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.509349:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.509366:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760727.509533:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 22867968<>22872064 00000080:00200000:1.0:1789760727.546327:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5583, 5583] 00000080:00200000:1.0:1789760727.546344:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.546350:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.546367:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22872064 i_blocks 45063 00000020:00200000:1.0:1789760727.546371:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.546382:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.546450:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22872064, count: 4096 00000080:00200000:1.0:1789760727.546458:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22872064, bytes: 4096 00000080:00200000:1.0:1789760727.546534:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.546753:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.546756:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22872064, 22876160) 0 0 00000080:00200000:1.0:1789760727.546761:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.546779:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5584, 5584] 00020000:00200000:1.0:1789760727.546789:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22872064 chunk: [22872064, 22876160) 22876160, 4096 00020000:00200000:1.0:1789760727.546794:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.546815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5584, 5584] 00000008:00010000:1.0:1789760727.546835:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.546841:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760727.546852:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22872064, 22876160) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.546866:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22872064=0x15d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.546933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.546936:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22876160 00000020:00010000:1.0:1789760727.546944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.546947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.547239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.547244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.547247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.547261:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.547267:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.547293:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760727.547459:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 22872064<>22876160 00000080:00200000:2.0:1789760727.576170:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5584, 5584] 00000080:00200000:2.0:1789760727.576191:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.576198:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.576215:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22876160 i_blocks 45063 00000020:00200000:2.0:1789760727.576220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.576232:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.576269:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22876160, count: 4096 00000080:00200000:2.0:1789760727.576277:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22876160, bytes: 4096 00000080:00200000:2.0:1789760727.576290:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.576299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.576301:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22876160, 22880256) 0 0 00000080:00200000:2.0:1789760727.576306:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.576325:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5585, 5585] 00020000:00200000:2.0:1789760727.576334:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22876160 chunk: [22876160, 22880256) 22880256, 4096 00020000:00200000:2.0:1789760727.576338:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.576358:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5585, 5585] 00000008:00010000:2.0:1789760727.576377:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.576383:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.576426:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22876160, 22880256) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.576456:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22876160=0x15d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.576521:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.576523:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22880256 00000020:00010000:2.0:1789760727.576533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.576536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.576539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.576542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.576545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.576553:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.576560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.576579:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760727.576693:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 22876160<>22880256 00000100:00100000:3.0:1789760727.577570:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.608617:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972904 gen 3 00000008:00010000:3.0:1789760727.608682:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760727.608693:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.608706:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.608728:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760727.608760:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.608761:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5585, 5585] 00000080:00200000:2.0:1789760727.608782:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.608788:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.608805:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22880256 i_blocks 45063 00000020:00200000:2.0:1789760727.608809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.608833:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.608876:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22880256, count: 4096 00000080:00200000:2.0:1789760727.608884:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22880256, bytes: 4096 00000080:00200000:2.0:1789760727.608894:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.608904:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.608907:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22880256, 22884352) 0 0 00000080:00200000:2.0:1789760727.608912:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.608930:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5586, 5586] 00020000:00200000:2.0:1789760727.608940:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22880256 chunk: [22880256, 22884352) 22884352, 4096 00020000:00200000:2.0:1789760727.608944:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.608965:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5586, 5586] 00000008:00010000:2.0:1789760727.608984:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.608990:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.609003:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22880256, 22884352) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.609019:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22880256=0x15d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.609088:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.609091:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22884352 00000020:00010000:2.0:1789760727.609100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.609104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.609107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.609111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.609113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.609122:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.609129:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.609146:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760727.609247:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 22880256<>22884352 00000100:00100000:3.0:1789760727.609554:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493696512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.648734:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972905 gen 3 00000008:00010000:3.0:1789760727.648799:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760727.648810:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.650940:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.651041:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760727.651079:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493696512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.651283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5586, 5586] 00000080:00200000:2.0:1789760727.651300:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.651305:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.651322:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22884352 i_blocks 45063 00000020:00200000:2.0:1789760727.651327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.651338:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.651373:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22884352, count: 4096 00000080:00200000:2.0:1789760727.651381:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22884352, bytes: 4096 00000080:00200000:2.0:1789760727.651702:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.651714:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.651716:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22884352, 22888448) 0 0 00000080:00200000:2.0:1789760727.651722:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.651742:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5587, 5587] 00020000:00200000:2.0:1789760727.651751:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22884352 chunk: [22884352, 22888448) 22888448, 4096 00020000:00200000:2.0:1789760727.651756:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.651776:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5587, 5587] 00000008:00010000:2.0:1789760727.651794:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.651800:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.651812:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22884352, 22888448) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.651953:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22884352=0x15d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.652025:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.652028:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22888448 00000020:00010000:2.0:1789760727.652036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.652040:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.652043:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.652047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.652050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.652058:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.652065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.652086:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760727.652198:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 22884352<>22888448 00000100:00100000:3.0:1789760727.652286:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.694624:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972906 gen 3 00000008:00010000:3.0:1789760727.694784:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760727.694793:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.694804:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.694824:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760727.694853:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.695017:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5587, 5587] 00000080:00200000:2.0:1789760727.695033:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.695038:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.695054:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22888448 i_blocks 45063 00000020:00200000:2.0:1789760727.695057:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.695067:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.695099:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22888448, count: 4096 00000080:00200000:2.0:1789760727.695105:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22888448, bytes: 4096 00000080:00200000:2.0:1789760727.695114:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.695122:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.695124:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22888448, 22892544) 0 0 00000080:00200000:2.0:1789760727.695128:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.695143:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5588, 5588] 00020000:00200000:2.0:1789760727.695151:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22888448 chunk: [22888448, 22892544) 22892544, 4096 00020000:00200000:2.0:1789760727.695156:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.695175:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5588, 5588] 00000008:00010000:2.0:1789760727.695192:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.695199:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.695213:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22888448, 22892544) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.695228:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22888448=0x15d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.695291:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.695294:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22892544 00000020:00010000:2.0:1789760727.695303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.695306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.695309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.695312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.695315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.695323:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.695330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.695357:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760727.696031:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 22888448<>22892544 00000100:00100000:3.0:1789760727.696320:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493696768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.766984:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972907 gen 3 00000008:00010000:3.0:1789760727.767039:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760727.767048:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.767060:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.767150:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760727.767179:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493696768:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.767258:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5588, 5588] 00000080:00200000:2.0:1789760727.767274:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.767280:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.767297:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22892544 i_blocks 45063 00000020:00200000:2.0:1789760727.767300:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.767310:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.767346:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22892544, count: 4096 00000080:00200000:2.0:1789760727.767353:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22892544, bytes: 4096 00000080:00200000:2.0:1789760727.767363:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.767372:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.767374:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22892544, 22896640) 0 0 00000080:00200000:2.0:1789760727.767378:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.767426:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5589, 5589] 00020000:00200000:2.0:1789760727.767445:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22892544 chunk: [22892544, 22896640) 22896640, 4096 00020000:00200000:2.0:1789760727.767449:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.767469:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5589, 5589] 00000008:00010000:2.0:1789760727.767549:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.767556:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.767569:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22892544, 22896640) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.767582:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22892544=0x15d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.767635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.767637:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22896640 00000020:00010000:2.0:1789760727.767645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.767649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.767652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.767655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.767658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.767665:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.767671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.767686:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760727.767797:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 22892544<>22896640 00000100:00100000:3.0:1789760727.767862:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696896:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760727.810811:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760727.810826:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.810839:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.810860:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760727.810898:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493696896:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.811966:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5589, 5589] 00000080:00200000:2.0:1789760727.811988:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.811994:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.812012:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22896640 i_blocks 45063 00000020:00200000:2.0:1789760727.812017:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.812032:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.812069:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22896640, count: 4096 00000080:00200000:2.0:1789760727.812077:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22896640, bytes: 4096 00000080:00200000:2.0:1789760727.812091:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.812100:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.812103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22896640, 22900736) 0 0 00000080:00200000:2.0:1789760727.812107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.812126:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5590, 5590] 00020000:00200000:2.0:1789760727.812134:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22896640 chunk: [22896640, 22900736) 22900736, 4096 00020000:00200000:2.0:1789760727.812139:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.812158:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5590, 5590] 00000008:00010000:2.0:1789760727.812177:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.812183:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.812197:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22896640, 22900736) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.812211:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22896640=0x15d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.812269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.812272:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22900736 00000020:00010000:2.0:1789760727.812281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.812285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.812288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.812292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.812294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.812302:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.812310:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.812328:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760727.812480:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 22896640<>22900736 00000100:00100000:2.0:1789760727.812528:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760727.812550:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.862265:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972908 gen 3 00000100:00100000:3.0:1789760727.862313:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80700 x1876699493697024/t4294972909(4294972909) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760743 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760727.862527:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760727.862535:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.862546:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.862573:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760727.862600:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697024:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760727.898802:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5591, 5591] 00000080:00200000:3.0:1789760727.898820:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.898825:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.898841:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22904832 i_blocks 45063 00000020:00200000:3.0:1789760727.898846:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.898857:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.898891:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22904832, count: 4096 00000080:00200000:3.0:1789760727.898899:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22904832, bytes: 4096 00000080:00200000:3.0:1789760727.898909:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.898929:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.898932:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22904832, 22908928) 0 0 00000080:00200000:3.0:1789760727.898936:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.898953:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5592, 5592] 00020000:00200000:3.0:1789760727.898962:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22904832 chunk: [22904832, 22908928) 22908928, 4096 00020000:00200000:3.0:1789760727.898967:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.898987:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5592, 5592] 00000008:00010000:3.0:1789760727.899005:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.899011:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760727.899023:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22904832, 22908928) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.899038:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22904832=0x15d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.899095:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.899098:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22908928 00000020:00010000:3.0:1789760727.899107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.899111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.899114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.899118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.899121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.899128:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.899135:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.899150:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760727.899249:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22904832<>22908928 00000100:00100000:3.0:1789760727.899294:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760727.899310:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493697280:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760727.934788:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972911 gen 3 00000008:00010000:3.0:1789760727.934866:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760727.934875:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760727.934886:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760727.934908:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760727.935329:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493697280:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.936374:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493697408:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760727.971732:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972912 gen 3 00000008:00010000:1.0:1789760727.971780:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760727.971789:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.971800:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.971822:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760727.971850:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493697408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760727.972002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5593, 5593] 00000080:00200000:2.0:1789760727.972019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.972026:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.972044:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22913024 i_blocks 45063 00000020:00200000:2.0:1789760727.972048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.972059:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.972097:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22913024, count: 4096 00000080:00200000:2.0:1789760727.972104:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22913024, bytes: 4096 00000080:00200000:2.0:1789760727.972115:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.972124:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.972126:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22913024, 22917120) 0 0 00000080:00200000:2.0:1789760727.972131:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.972150:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5594, 5594] 00020000:00200000:2.0:1789760727.972160:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22913024 chunk: [22913024, 22917120) 22917120, 4096 00020000:00200000:2.0:1789760727.972166:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.972186:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5594, 5594] 00000008:00010000:2.0:1789760727.972203:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.972208:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760727.972220:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22913024, 22917120) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.972234:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22913024=0x15da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.972290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.972293:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22917120 00000020:00010000:2.0:1789760727.972301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.972305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.972307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.972310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.972313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.972320:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.972326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.972341:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760727.972512:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 22913024<>22917120 00000100:00100000:2.0:1789760727.972553:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760727.972570:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697536:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.003278:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972913 gen 3 00000008:00010000:3.0:1789760728.003334:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760728.003343:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.003355:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.003466:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760728.003495:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760728.003660:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5594, 5594] 00000080:00200000:2.0:1789760728.003677:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760728.003681:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760728.003695:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22917120 i_blocks 45063 00000020:00200000:2.0:1789760728.003698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760728.003708:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760728.003742:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22917120, count: 4096 00000080:00200000:2.0:1789760728.003750:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22917120, bytes: 4096 00000080:00200000:2.0:1789760728.003760:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760728.003769:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760728.003771:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22917120, 22921216) 0 0 00000080:00200000:2.0:1789760728.003775:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760728.003789:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5595, 5595] 00020000:00200000:2.0:1789760728.003797:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22917120 chunk: [22917120, 22921216) 22921216, 4096 00020000:00200000:2.0:1789760728.003801:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760728.003860:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5595, 5595] 00000008:00010000:2.0:1789760728.003877:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760728.003882:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760728.003893:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22917120, 22921216) append 0 append lockpos 0 00000080:00200000:2.0:1789760728.003906:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22917120=0x15db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760728.004003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760728.004006:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22921216 00000020:00010000:2.0:1789760728.004015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760728.004019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.004022:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.004025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760728.004028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760728.004036:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760728.004042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760728.004058:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760728.004157:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 22917120<>22921216 00000100:00100000:3.0:1789760728.004210:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.052806:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972914 gen 3 00000008:00010000:3.0:1789760728.052857:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760728.052865:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.052877:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.052941:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760728.052972:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.053786:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493697792:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.113074:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972915 gen 3 00000008:00010000:1.0:1789760728.113120:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760728.113128:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.113139:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.113159:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760728.113186:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493697792:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760728.114647:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5596, 5596] 00000080:00200000:3.0:1789760728.114669:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.114675:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.114734:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22925312 i_blocks 45063 00000020:00200000:3.0:1789760728.114741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.114753:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.114787:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22925312, count: 4096 00000080:00200000:3.0:1789760728.114795:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22925312, bytes: 4096 00000080:00200000:3.0:1789760728.114805:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.114815:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.114817:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22925312, 22929408) 0 0 00000080:00200000:3.0:1789760728.114821:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.114838:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5597, 5597] 00020000:00200000:3.0:1789760728.114848:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22925312 chunk: [22925312, 22929408) 22929408, 4096 00020000:00200000:3.0:1789760728.114852:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.114872:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5597, 5597] 00000008:00010000:3.0:1789760728.114889:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.114895:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760728.114906:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22925312, 22929408) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.114920:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22925312=0x15dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.114981:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.114984:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22929408 00000020:00010000:3.0:1789760728.114994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.114997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.115000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.115003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.115006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.115015:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.115022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.115041:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760728.115146:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22925312<>22929408 00000100:00100000:3.0:1789760728.115197:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.116351:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.169365:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972916 gen 3 00000008:00010000:3.0:1789760728.169443:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760728.169450:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.169460:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.169541:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760728.169565:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493697920:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760728.169971:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5597, 5597] 00000080:00200000:1.0:1789760728.169985:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760728.169991:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760728.170002:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22929408 i_blocks 45063 00000020:00200000:1.0:1789760728.170006:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.170015:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.170043:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22929408, count: 4096 00000080:00200000:1.0:1789760728.170048:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22929408, bytes: 4096 00000080:00200000:1.0:1789760728.170057:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.170065:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.170067:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22929408, 22933504) 0 0 00000080:00200000:1.0:1789760728.170071:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.170085:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5598, 5598] 00020000:00200000:1.0:1789760728.170092:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22929408 chunk: [22929408, 22933504) 22933504, 4096 00020000:00200000:1.0:1789760728.170095:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.170111:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5598, 5598] 00000008:00010000:1.0:1789760728.170125:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.170130:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.170139:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22929408, 22933504) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.170150:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22929408=0x15de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.170198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.170201:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22933504 00000020:00010000:1.0:1789760728.170209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.170211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.170214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.170217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.170219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.170227:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.170232:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760728.170249:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760728.170335:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 22929408<>22933504 00000100:00100000:1.0:1789760728.212860:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972917 gen 3 00000008:00010000:1.0:1789760728.212907:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760728.212914:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.212924:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.212941:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760728.212963:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493698048:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760728.213064:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5598, 5598] 00000080:00200000:3.0:1789760728.213083:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.213087:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.213097:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22933504 i_blocks 45063 00000020:00200000:3.0:1789760728.213099:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.213106:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.213127:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22933504, count: 4096 00000080:00200000:3.0:1789760728.213132:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22933504, bytes: 4096 00000080:00200000:3.0:1789760728.213138:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.213144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.213145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22933504, 22937600) 0 0 00000080:00200000:3.0:1789760728.213148:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.213158:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5599, 5599] 00020000:00200000:3.0:1789760728.213164:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22933504 chunk: [22933504, 22937600) 22937600, 4096 00020000:00200000:3.0:1789760728.213166:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.213178:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5599, 5599] 00000008:00010000:3.0:1789760728.213188:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.213191:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760728.213198:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22933504, 22937600) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.213207:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22933504=0x15df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.213242:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.213244:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22937600 00000020:00010000:3.0:1789760728.213249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.213251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.213253:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.213255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.213257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.213262:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.213265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.213275:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760728.213340:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22933504<>22937600 00000100:00100000:2.0:1789760728.213389:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493698176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.256090:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972918 gen 3 00000008:00010000:3.0:1789760728.256140:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760728.256148:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.256159:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.256182:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760728.256221:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493698176:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760728.256739:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5599, 5599] 00000080:00200000:1.0:1789760728.256757:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760728.256763:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760728.256780:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22937600 i_blocks 45063 00000020:00200000:1.0:1789760728.256784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.256795:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.256831:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22937600, count: 4096 00000080:00200000:1.0:1789760728.256839:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22937600, bytes: 4096 00000080:00200000:1.0:1789760728.256850:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.256860:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.256862:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22937600, 22941696) 0 0 00000080:00200000:1.0:1789760728.256867:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.256883:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5600, 5600] 00020000:00200000:1.0:1789760728.256892:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22937600 chunk: [22937600, 22941696) 22941696, 4096 00020000:00200000:1.0:1789760728.256896:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.256920:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5600, 5600] 00000008:00010000:1.0:1789760728.257015:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.257025:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.257039:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22937600, 22941696) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.257054:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22937600=0x15e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.257112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.257115:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22941696 00000020:00010000:1.0:1789760728.257124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.257128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.257132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.257135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.257139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.257150:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.257157:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760728.257175:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760728.257735:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 22937600<>22941696 00000080:00200000:3.0:1789760728.293706:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5600, 5600] 00000080:00200000:3.0:1789760728.293722:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.293727:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.293741:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22941696 i_blocks 45063 00000020:00200000:3.0:1789760728.293744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.293752:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.293784:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22941696, count: 4096 00000080:00200000:3.0:1789760728.293790:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22941696, bytes: 4096 00000080:00200000:3.0:1789760728.293797:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.293804:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.293806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22941696, 22945792) 0 0 00000080:00200000:3.0:1789760728.293810:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.293825:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5601, 5601] 00020000:00200000:3.0:1789760728.293833:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22941696 chunk: [22941696, 22945792) 22945792, 4096 00020000:00200000:3.0:1789760728.293837:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.293857:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5601, 5601] 00000008:00010000:3.0:1789760728.293875:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.293881:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760728.293897:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22941696, 22945792) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.293912:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22941696=0x15e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.294045:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.294048:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22945792 00000020:00010000:3.0:1789760728.294057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.294062:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.294064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.294067:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.294070:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.294078:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.294086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.294102:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760728.294213:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22941696<>22945792 00000100:00100000:3.0:1789760728.294344:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.294363:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493698432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.343074:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972920 gen 3 00000008:00010000:3.0:1789760728.343121:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760728.343129:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.343139:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.343198:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760728.343234:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493698432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760728.343820:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5601, 5601] 00000080:00200000:2.0:1789760728.343839:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760728.343845:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760728.343861:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22945792 i_blocks 45063 00000020:00200000:2.0:1789760728.343865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760728.343876:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760728.343921:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22945792, count: 4096 00000080:00200000:2.0:1789760728.343928:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22945792, bytes: 4096 00000080:00200000:2.0:1789760728.343947:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760728.343957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760728.343960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22945792, 22949888) 0 0 00000080:00200000:2.0:1789760728.343964:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760728.343979:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5602, 5602] 00020000:00200000:2.0:1789760728.343987:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22945792 chunk: [22945792, 22949888) 22949888, 4096 00020000:00200000:2.0:1789760728.343992:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760728.344011:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5602, 5602] 00000008:00010000:2.0:1789760728.344030:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760728.344036:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760728.344047:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22945792, 22949888) append 0 append lockpos 0 00000080:00200000:2.0:1789760728.344063:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22945792=0x15e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760728.344120:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760728.344124:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22949888 00000020:00010000:2.0:1789760728.344132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760728.344136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.344139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f977e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.344143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760728.344146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760728.344153:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760728.344161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760728.344180:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760728.344295:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 22945792<>22949888 00000100:00100000:2.0:1789760728.344338:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760728.344357:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493698560:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.401279:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972921 gen 3 00000008:00010000:3.0:1789760728.401328:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760728.401336:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.401347:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.401523:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760728.401553:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493698560:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.402519:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760728.402544:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493698688:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760728.462031:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760728.462046:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.462056:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.462079:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760728.462109:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493698688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.464807:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493698816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.508282:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972922 gen 3 00000100:00100000:1.0:1789760728.508326:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304e300 x1876699493698816/t4294972923(4294972923) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760744 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760728.508374:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760728.508381:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.508391:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.508509:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760728.508533:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493698816:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760728.508781:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22958080 i_blocks 45063 00000020:00200000:1.0:1789760728.508785:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.508796:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.508829:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22958080, count: 4096 00000080:00200000:1.0:1789760728.508836:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22958080, bytes: 4096 00000080:00200000:1.0:1789760728.508846:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.508855:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.508857:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22958080, 22962176) 0 0 00000080:00200000:1.0:1789760728.508862:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.508878:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5605, 5605] 00020000:00200000:1.0:1789760728.508887:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22958080 chunk: [22958080, 22962176) 22962176, 4096 00020000:00200000:1.0:1789760728.508892:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.508909:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5605, 5605] 00000008:00010000:1.0:1789760728.508925:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.508930:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.508942:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22958080, 22962176) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.508955:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22958080=0x15e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.509007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.509010:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22962176 00000020:00010000:1.0:1789760728.509085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.509089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.509092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.509096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.509099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.509110:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.509116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760728.509129:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760728.509226:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 22958080<>22962176 00000100:00100000:1.0:1789760728.554030:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972924 gen 3 00000008:00010000:1.0:1789760728.554098:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:1.0:1789760728.554106:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.554117:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.554201:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:1.0:1789760728.554233:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493698944:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760728.554836:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5605, 5605] 00000080:00200000:3.0:1789760728.554853:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.554860:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.554879:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22962176 i_blocks 45063 00000020:00200000:3.0:1789760728.554883:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.554896:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.554931:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22962176, count: 4096 00000080:00200000:3.0:1789760728.554939:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22962176, bytes: 4096 00000080:00200000:3.0:1789760728.554953:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.554962:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.554965:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22962176, 22966272) 0 0 00000080:00200000:3.0:1789760728.554969:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.554985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5606, 5606] 00020000:00200000:3.0:1789760728.554994:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22962176 chunk: [22962176, 22966272) 22966272, 4096 00020000:00200000:3.0:1789760728.554999:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.555018:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5606, 5606] 00000008:00010000:3.0:1789760728.555036:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.555044:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760728.555055:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22962176, 22966272) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.555068:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22962176=0x15e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.555127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.555130:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22966272 00000020:00010000:3.0:1789760728.555139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.555143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.555146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.555150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.555154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.555162:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.555168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.555187:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760728.555298:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22962176<>22966272 00000100:00100000:2.0:1789760728.555775:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493699072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.602936:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972925 gen 3 00000008:00010000:3.0:1789760728.603001:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760728.603011:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.603024:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.603050:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760728.603454:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493699072:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.604328:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699200:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.652489:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972926 gen 3 00000008:00010000:1.0:1789760728.652538:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760728.652547:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.652558:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.652579:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760728.653189:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699200:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760728.653234:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:1.0:1789760728.653335:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 22970368<>22974464 00000100:00100000:1.0:1789760728.653374:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760728.653386:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699328:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.728089:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972927 gen 3 00000008:00010000:1.0:1789760728.728142:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760728.728157:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.728168:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.728251:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760728.728282:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699328:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.731264:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.762790:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972928 gen 3 00000008:00010000:1.0:1789760728.762840:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760728.762848:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.762860:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.762887:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760728.762928:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493699456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.763806:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493699584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760728.806581:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972929 gen 3 00000008:00010000:1.0:1789760728.806632:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760728.806641:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.806653:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.806742:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760728.806777:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493699584:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760728.842080:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5611, 5611] 00000080:00200000:3.0:1789760728.842097:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.842103:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.842118:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22986752 i_blocks 45063 00000020:00200000:3.0:1789760728.842123:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.842134:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.842166:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22986752, count: 4096 00000080:00200000:3.0:1789760728.842174:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22986752, bytes: 4096 00000080:00200000:3.0:1789760728.842185:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.842194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.842197:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22986752, 22990848) 0 0 00000080:00200000:3.0:1789760728.842202:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.842219:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5612, 5612] 00020000:00200000:3.0:1789760728.842227:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22986752 chunk: [22986752, 22990848) 22990848, 4096 00020000:00200000:3.0:1789760728.842232:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.842251:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5612, 5612] 00000008:00010000:3.0:1789760728.842268:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.842275:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760728.842287:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22986752, 22990848) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.842302:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22986752=0x15ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.842360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.842363:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22990848 00000020:00010000:3.0:1789760728.842371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.842375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.842378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.842382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.842385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.842442:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.842450:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.842468:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760728.842587:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 22986752<>22990848 00000100:00100000:3.0:1789760728.842635:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760728.842654:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493699840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760728.877525:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972931 gen 3 00000008:00010000:3.0:1789760728.877593:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760728.877602:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760728.877613:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760728.877638:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760728.877667:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493699840:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760728.922525:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5613, 5613] 00000080:00200000:1.0:1789760728.922542:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760728.922548:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760728.922564:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22994944 i_blocks 45063 00000020:00200000:1.0:1789760728.922567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.922579:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.922613:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22994944, count: 4096 00000080:00200000:1.0:1789760728.922621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22994944, bytes: 4096 00000080:00200000:1.0:1789760728.922632:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.922642:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.922644:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22994944, 22999040) 0 0 00000080:00200000:1.0:1789760728.922648:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.922665:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5614, 5614] 00020000:00200000:1.0:1789760728.922673:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22994944 chunk: [22994944, 22999040) 22999040, 4096 00020000:00200000:1.0:1789760728.922677:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.922695:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5614, 5614] 00000008:00010000:1.0:1789760728.922713:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.922720:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.922732:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22994944, 22999040) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.922746:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22994944=0x15ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.922801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.922804:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22999040 00000020:00010000:1.0:1789760728.922827:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.922831:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.922834:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.922838:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.922841:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.922851:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.922858:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.952501:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5614, 5614] 00000080:00200000:1.0:1789760728.952520:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760728.952527:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760728.952543:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22999040 i_blocks 45063 00000020:00200000:1.0:1789760728.952547:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.952557:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.952591:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22999040, count: 4096 00000080:00200000:1.0:1789760728.952599:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22999040, bytes: 4096 00000080:00200000:1.0:1789760728.952608:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.952617:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.952619:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22999040, 23003136) 0 0 00000080:00200000:1.0:1789760728.952624:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.952640:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5615, 5615] 00020000:00200000:1.0:1789760728.952648:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22999040 chunk: [22999040, 23003136) 23003136, 4096 00020000:00200000:1.0:1789760728.952652:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.952750:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5615, 5615] 00000008:00010000:1.0:1789760728.952769:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.952775:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.952786:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22999040, 23003136) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.952800:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=22999040=0x15ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.952857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.952860:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23003136 00000020:00010000:1.0:1789760728.952868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.952871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.952876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.952879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.952882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.952893:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.952900:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760728.953133:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760728.953236:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 22999040<>23003136 00000080:00200000:1.0:1789760728.992838:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5615, 5615] 00000080:00200000:1.0:1789760728.992854:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760728.992860:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760728.992875:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23003136 i_blocks 45063 00000020:00200000:1.0:1789760728.992879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760728.992889:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760728.992920:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23003136, count: 4096 00000080:00200000:1.0:1789760728.992927:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23003136, bytes: 4096 00000080:00200000:1.0:1789760728.992937:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760728.992945:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760728.992948:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23003136, 23007232) 0 0 00000080:00200000:1.0:1789760728.992952:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760728.992968:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5616, 5616] 00020000:00200000:1.0:1789760728.992977:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23003136 chunk: [23003136, 23007232) 23007232, 4096 00020000:00200000:1.0:1789760728.992981:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760728.992999:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5616, 5616] 00000008:00010000:1.0:1789760728.993018:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760728.993024:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760728.993035:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23003136, 23007232) append 0 append lockpos 0 00000080:00200000:1.0:1789760728.993048:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23003136=0x15f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760728.993104:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760728.993107:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23007232 00000020:00010000:1.0:1789760728.993114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760728.993118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.993123:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760728.993126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760728.993129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760728.993139:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760728.993147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760728.993161:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e926c0 00000008:00100000:1.0:1789760728.993260:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 23003136<>23007232 00000080:00200000:1.0:1789760729.042439:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5616, 5616] 00000080:00200000:1.0:1789760729.042455:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.042461:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.042476:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23007232 i_blocks 45063 00000020:00200000:1.0:1789760729.042480:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.042489:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.042522:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23007232, count: 4096 00000080:00200000:1.0:1789760729.042528:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23007232, bytes: 4096 00000080:00200000:1.0:1789760729.042537:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.042546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.042548:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23007232, 23011328) 0 0 00000080:00200000:1.0:1789760729.042553:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.042568:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5617, 5617] 00020000:00200000:1.0:1789760729.042576:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23007232 chunk: [23007232, 23011328) 23011328, 4096 00020000:00200000:1.0:1789760729.042581:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.042598:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5617, 5617] 00000008:00010000:1.0:1789760729.042614:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.042620:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.042631:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23007232, 23011328) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.042649:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23007232=0x15f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.042699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.042702:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23011328 00000020:00010000:1.0:1789760729.042709:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.042712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.042716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.042719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.042722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.042732:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.042738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.042752:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760729.043010:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 23007232<>23011328 00000080:00200000:1.0:1789760729.099831:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5617, 5617] 00000080:00200000:1.0:1789760729.099853:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.099859:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.099875:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23011328 i_blocks 45063 00000020:00200000:1.0:1789760729.099879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.099891:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.099926:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23011328, count: 4096 00000080:00200000:1.0:1789760729.099933:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23011328, bytes: 4096 00000080:00200000:1.0:1789760729.099948:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.099957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.099959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23011328, 23015424) 0 0 00000080:00200000:1.0:1789760729.099963:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.099979:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5618, 5618] 00020000:00200000:1.0:1789760729.099989:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23011328 chunk: [23011328, 23015424) 23015424, 4096 00020000:00200000:1.0:1789760729.100004:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.100027:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5618, 5618] 00000008:00010000:1.0:1789760729.100047:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.100054:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.100065:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23011328, 23015424) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.100080:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23011328=0x15f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.100146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.100149:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23015424 00000020:00010000:1.0:1789760729.100157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.100161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.100165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.100169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.100173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.100186:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.100193:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.100212:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760729.100324:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 23011328<>23015424 00000080:00200000:1.0:1789760729.140613:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5618, 5618] 00000080:00200000:1.0:1789760729.140630:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.140635:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.140651:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23015424 i_blocks 45063 00000020:00200000:1.0:1789760729.140656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.140665:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.140712:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23015424, count: 4096 00000080:00200000:1.0:1789760729.140719:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23015424, bytes: 4096 00000080:00200000:1.0:1789760729.140729:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.140738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.140741:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23015424, 23019520) 0 0 00000080:00200000:1.0:1789760729.140746:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.140761:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5619, 5619] 00020000:00200000:1.0:1789760729.140874:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23015424 chunk: [23015424, 23019520) 23019520, 4096 00020000:00200000:1.0:1789760729.140880:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.140902:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5619, 5619] 00000008:00010000:1.0:1789760729.140991:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.140997:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.141010:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23015424, 23019520) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.141025:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23015424=0x15f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.141082:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.141085:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23019520 00000020:00010000:1.0:1789760729.141094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.141097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.141102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.141106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.141109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.141119:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.141125:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.141140:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760729.141237:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 23015424<>23019520 00000080:00200000:1.0:1789760729.210611:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5619, 5619] 00000080:00200000:1.0:1789760729.210628:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.210635:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.210651:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23019520 i_blocks 45063 00000020:00200000:1.0:1789760729.210655:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.210667:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.210703:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23019520, count: 4096 00000080:00200000:1.0:1789760729.210712:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23019520, bytes: 4096 00000080:00200000:1.0:1789760729.210721:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.210730:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.210732:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23019520, 23023616) 0 0 00000080:00200000:1.0:1789760729.210737:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.210752:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5620, 5620] 00020000:00200000:1.0:1789760729.210761:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23019520 chunk: [23019520, 23023616) 23023616, 4096 00020000:00200000:1.0:1789760729.210766:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.212207:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5620, 5620] 00000008:00010000:1.0:1789760729.212232:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.212238:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.212251:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23019520, 23023616) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.212267:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23019520=0x15f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.212330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.212334:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23023616 00000020:00010000:1.0:1789760729.212342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.212346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.212349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.212353:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.212356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.212368:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.212376:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.212481:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760729.212651:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 23019520<>23023616 00000080:00200000:1.0:1789760729.283529:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5620, 5620] 00000080:00200000:1.0:1789760729.283545:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.283551:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.283567:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23023616 i_blocks 45063 00000020:00200000:1.0:1789760729.283570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.283586:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.283620:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23023616, count: 4096 00000080:00200000:1.0:1789760729.283627:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23023616, bytes: 4096 00000080:00200000:1.0:1789760729.283645:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.283654:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.283656:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23023616, 23027712) 0 0 00000080:00200000:1.0:1789760729.283661:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.283676:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5621, 5621] 00020000:00200000:1.0:1789760729.283687:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23023616 chunk: [23023616, 23027712) 23027712, 4096 00020000:00200000:1.0:1789760729.283692:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.283712:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5621, 5621] 00000008:00010000:1.0:1789760729.283728:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.283733:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.283744:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23023616, 23027712) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.283758:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23023616=0x15f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.283826:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.283830:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23027712 00000020:00010000:1.0:1789760729.283839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.283842:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.283846:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.283850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.283853:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.283864:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.283871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.283892:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760729.284013:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 23023616<>23027712 00000080:00200000:1.0:1789760729.352616:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5621, 5621] 00000080:00200000:1.0:1789760729.352643:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.352650:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.352668:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23027712 i_blocks 45063 00000020:00200000:1.0:1789760729.352673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.352688:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.352724:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23027712, count: 4096 00000080:00200000:1.0:1789760729.352732:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23027712, bytes: 4096 00000080:00200000:1.0:1789760729.352747:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.352764:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.352767:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23027712, 23031808) 0 0 00000080:00200000:1.0:1789760729.352772:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.352789:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5622, 5622] 00020000:00200000:1.0:1789760729.352797:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23027712 chunk: [23027712, 23031808) 23031808, 4096 00020000:00200000:1.0:1789760729.352802:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.352824:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5622, 5622] 00000008:00010000:1.0:1789760729.352843:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.352849:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.352861:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23027712, 23031808) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.352877:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23027712=0x15f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.352944:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.352947:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23031808 00000020:00010000:1.0:1789760729.352956:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.352959:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.352962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.352966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.352969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.352981:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.352988:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.353015:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760729.353130:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 23027712<>23031808 00000100:00100000:1.0:1789760729.353188:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760729.353349:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493701120:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760729.423891:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760729.423907:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.423919:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.423948:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760729.423978:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493701120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760729.424083:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5622, 5622] 00000080:00200000:3.0:1789760729.424100:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.424106:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.424121:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23031808 i_blocks 45063 00000020:00200000:3.0:1789760729.424125:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.424135:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.424169:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23031808, count: 4096 00000080:00200000:3.0:1789760729.424175:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23031808, bytes: 4096 00000080:00200000:3.0:1789760729.424184:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.424192:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.424195:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23031808, 23035904) 0 0 00000080:00200000:3.0:1789760729.424199:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.424216:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5623, 5623] 00020000:00200000:3.0:1789760729.424225:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23031808 chunk: [23031808, 23035904) 23035904, 4096 00020000:00200000:3.0:1789760729.424229:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.424248:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5623, 5623] 00000008:00010000:3.0:1789760729.424270:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.424276:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760729.424288:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23031808, 23035904) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.424303:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23031808=0x15f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.424366:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.424370:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23035904 00000020:00010000:3.0:1789760729.424378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.424381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.424384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.424388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.424391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.424503:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.424511:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.424527:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760729.424639:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 23031808<>23035904 00000100:00100000:2.0:1789760729.424744:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760729.424765:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760729.487289:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972941 gen 3 00000100:00100000:3.0:1789760729.487335:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148380 x1876699493701248/t4294972942(4294972942) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760745 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760729.487385:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760729.487432:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.487443:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.487468:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760729.487490:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701248:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760729.487675:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5623, 5623] 00000080:00200000:1.0:1789760729.487692:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.487698:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.487712:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23035904 i_blocks 45063 00000020:00200000:1.0:1789760729.487716:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.487725:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.487761:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23035904, count: 4096 00000080:00200000:1.0:1789760729.487768:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23035904, bytes: 4096 00000080:00200000:1.0:1789760729.487781:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.487788:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.487790:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23035904, 23040000) 0 0 00000080:00200000:1.0:1789760729.487795:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.487810:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5624, 5624] 00020000:00200000:1.0:1789760729.487819:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23035904 chunk: [23035904, 23040000) 23040000, 4096 00020000:00200000:1.0:1789760729.487824:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.487843:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5624, 5624] 00000008:00010000:1.0:1789760729.487862:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.487867:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.487878:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23035904, 23040000) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.487891:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23035904=0x15f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.488256:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.488259:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23040000 00000020:00010000:1.0:1789760729.488269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.488273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.488276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.488280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.488283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.488294:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.488301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.488318:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760729.488537:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 23035904<>23040000 00000080:00200000:3.0:1789760729.574050:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5624, 5624] 00000080:00200000:3.0:1789760729.574134:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.574142:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.574161:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23040000 i_blocks 45063 00000020:00200000:3.0:1789760729.574167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.574187:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.574222:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23040000, count: 4096 00000080:00200000:3.0:1789760729.574230:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23040000, bytes: 4096 00000080:00200000:3.0:1789760729.574244:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.574255:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.574257:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23040000, 23044096) 0 0 00000080:00200000:3.0:1789760729.574263:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.574282:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5625, 5625] 00020000:00200000:3.0:1789760729.574291:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23040000 chunk: [23040000, 23044096) 23044096, 4096 00020000:00200000:3.0:1789760729.574296:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.574409:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5625, 5625] 00000008:00010000:3.0:1789760729.574439:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.574445:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760729.574458:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23040000, 23044096) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.574471:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23040000=0x15f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.575959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.575963:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23044096 00000020:00010000:3.0:1789760729.575975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.575978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.575982:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.575986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.575989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.576000:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.576010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.576040:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760729.576164:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 23040000<>23044096 00000100:00100000:2.0:1789760729.576641:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760729.649752:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972944 gen 3 00000008:00010000:3.0:1789760729.649805:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760729.649814:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.649825:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.649838:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760729.649868:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701504:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760729.688494:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5626, 5626] 00000080:00200000:1.0:1789760729.688511:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.688517:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.688533:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23048192 i_blocks 45063 00000020:00200000:1.0:1789760729.688536:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.688557:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.688591:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23048192, count: 4096 00000080:00200000:1.0:1789760729.688599:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23048192, bytes: 4096 00000080:00200000:1.0:1789760729.688608:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.688618:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.688620:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23048192, 23052288) 0 0 00000080:00200000:1.0:1789760729.688625:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.688641:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5627, 5627] 00020000:00200000:1.0:1789760729.688654:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23048192 chunk: [23048192, 23052288) 23052288, 4096 00020000:00200000:1.0:1789760729.688659:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.688678:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5627, 5627] 00000008:00010000:1.0:1789760729.688696:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.688702:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760729.688716:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23048192, 23052288) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.688730:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23048192=0x15fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.688787:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.688790:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23052288 00000020:00010000:1.0:1789760729.688799:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.688803:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.688807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.688810:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.688814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.688825:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.688832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.688846:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760729.688954:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 23048192<>23052288 00000080:00200000:3.0:1789760729.739658:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5627, 5627] 00000080:00200000:3.0:1789760729.739685:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.739692:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.739711:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23052288 i_blocks 45063 00000020:00200000:3.0:1789760729.739716:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.739735:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.739772:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23052288, count: 4096 00000080:00200000:3.0:1789760729.739781:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23052288, bytes: 4096 00000080:00200000:3.0:1789760729.739797:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.739806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.739809:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23052288, 23056384) 0 0 00000080:00200000:3.0:1789760729.739813:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.739829:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5628, 5628] 00020000:00200000:3.0:1789760729.739837:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23052288 chunk: [23052288, 23056384) 23056384, 4096 00020000:00200000:3.0:1789760729.739841:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.739862:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5628, 5628] 00000008:00010000:3.0:1789760729.739880:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.739887:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760729.739899:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23052288, 23056384) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.739913:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23052288=0x15fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.739977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.739980:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23056384 00000020:00010000:3.0:1789760729.739987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.739991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.739994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.739997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.740001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.740009:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.740016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.740040:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760729.740149:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 23052288<>23056384 00000100:00100000:3.0:1789760729.740200:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.740225:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701888:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760729.780706:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760729.780723:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.780734:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.780754:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760729.780784:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493701888:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760729.825220:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5629, 5629] 00000080:00200000:3.0:1789760729.825237:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760729.825243:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760729.825259:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23060480 i_blocks 45063 00000020:00200000:3.0:1789760729.825263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760729.825275:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760729.825308:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23060480, count: 4096 00000080:00200000:3.0:1789760729.825315:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23060480, bytes: 4096 00000080:00200000:3.0:1789760729.825324:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760729.825333:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760729.825336:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23060480, 23064576) 0 0 00000080:00200000:3.0:1789760729.825340:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760729.825355:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5630, 5630] 00020000:00200000:3.0:1789760729.825364:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23060480 chunk: [23060480, 23064576) 23064576, 4096 00020000:00200000:3.0:1789760729.825368:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760729.825386:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5630, 5630] 00000008:00010000:3.0:1789760729.825440:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760729.825446:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760729.825458:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23060480, 23064576) append 0 append lockpos 0 00000080:00200000:3.0:1789760729.825479:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23060480=0x15fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760729.825678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760729.825681:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23064576 00000020:00010000:3.0:1789760729.825690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760729.825694:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.825697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760729.825701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760729.825705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760729.825713:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760729.825721:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.825736:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760729.825839:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 23060480<>23064576 00000100:00100000:2.0:1789760729.826210:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760729.826231:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493702144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760729.862617:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972949 gen 3 00000008:00010000:3.0:1789760729.862670:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760729.862680:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.862691:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.862713:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760729.862742:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493702144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760729.862929:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5630, 5630] 00000080:00200000:2.0:1789760729.862946:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.862952:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.862969:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23064576 i_blocks 45063 00000020:00200000:2.0:1789760729.862973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.862982:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.863018:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23064576, count: 4096 00000080:00200000:2.0:1789760729.863025:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23064576, bytes: 4096 00000080:00200000:2.0:1789760729.863037:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.863046:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.863048:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23064576, 23068672) 0 0 00000080:00200000:2.0:1789760729.863053:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.863068:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5631, 5631] 00020000:00200000:2.0:1789760729.863077:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23064576 chunk: [23064576, 23068672) 23068672, 4096 00020000:00200000:2.0:1789760729.863082:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.863103:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5631, 5631] 00000008:00010000:2.0:1789760729.863121:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.863126:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760729.863138:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23064576, 23068672) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.863152:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23064576=0x15ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.863216:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.863218:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23068672 00000020:00010000:2.0:1789760729.863227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.863231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.863235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.863238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.863241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.863250:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.863256:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.863273:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760729.863381:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23064576<>23068672 00000100:00100000:2.0:1789760729.863470:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760729.863488:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760729.908366:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972950 gen 3 00000008:00010000:3.0:1789760729.908465:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760729.908474:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.908494:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.908510:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760729.908541:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760729.908681:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5631, 5631] 00000080:00200000:2.0:1789760729.908699:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.908705:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.908722:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23068672 i_blocks 45063 00000020:00200000:2.0:1789760729.908726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.908737:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.908769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23068672, count: 4096 00000080:00200000:2.0:1789760729.908777:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23068672, bytes: 4096 00000080:00200000:2.0:1789760729.908787:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.908795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.908798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23068672, 23072768) 0 0 00000080:00200000:2.0:1789760729.908803:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.908820:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5632, 5632] 00020000:00200000:2.0:1789760729.908829:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23068672 chunk: [23068672, 23072768) 23072768, 4096 00020000:00200000:2.0:1789760729.908834:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.908855:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5632, 5632] 00000008:00010000:2.0:1789760729.908874:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.908880:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760729.908892:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23068672, 23072768) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.908918:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23068672=0x1600000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.908982:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.908985:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23072768 00000020:00010000:2.0:1789760729.908993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.908997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.909000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.909004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.909007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.909015:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.909021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.909036:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760729.909134:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23068672<>23072768 00000100:00100000:3.0:1789760729.909198:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702400:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760729.951715:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760729.951729:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.951740:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.951762:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760729.951790:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702400:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760729.952100:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5632, 5632] 00000080:00200000:2.0:1789760729.952118:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.952124:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.952140:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23072768 i_blocks 45063 00000020:00200000:2.0:1789760729.952144:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.952155:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.952189:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23072768, count: 4096 00000080:00200000:2.0:1789760729.952196:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23072768, bytes: 4096 00000080:00200000:2.0:1789760729.952205:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.952214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.952217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23072768, 23076864) 0 0 00000080:00200000:2.0:1789760729.952221:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.952237:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5633, 5633] 00020000:00200000:2.0:1789760729.952245:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23072768 chunk: [23072768, 23076864) 23076864, 4096 00020000:00200000:2.0:1789760729.952249:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.952270:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5633, 5633] 00000008:00010000:2.0:1789760729.952289:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.952295:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760729.952308:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23072768, 23076864) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.952322:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23072768=0x1601000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.952375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.952378:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23076864 00000020:00010000:2.0:1789760729.952385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.952388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.952391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.952556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.952559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.952569:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.952577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760729.952634:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:3.0:1789760729.952734:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 23072768<>23076864 00000100:00100000:3.0:1789760729.952774:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.952792:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760729.982207:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972952 gen 3 00000008:00010000:3.0:1789760729.982270:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760729.982279:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760729.982290:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760729.982351:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760729.982380:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702528:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760729.982735:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5633, 5633] 00000080:00200000:2.0:1789760729.982752:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.982757:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.982775:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23076864 i_blocks 47111 00000020:00200000:2.0:1789760729.982779:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.982790:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.982868:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23076864, count: 4096 00000080:00200000:2.0:1789760729.982877:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23076864, bytes: 4096 00000080:00200000:2.0:1789760729.982888:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.982959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.982962:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23076864, 23080960) 0 0 00000080:00200000:2.0:1789760729.982968:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.982985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5634, 5634] 00020000:00200000:2.0:1789760729.982993:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23076864 chunk: [23076864, 23080960) 23080960, 4096 00020000:00200000:2.0:1789760729.982997:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.983017:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5634, 5634] 00000008:00010000:2.0:1789760729.983035:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.983041:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760729.983053:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23076864, 23080960) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.983066:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23076864=0x1602000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.983123:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.983126:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23080960 00000020:00010000:2.0:1789760729.983133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.983137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.983140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.983143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.983146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.983155:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.983162:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.983179:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760729.983279:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23076864<>23080960 00000100:00100000:3.0:1789760729.983347:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760730.013701:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972953 gen 3 00000008:00010000:3.0:1789760730.013752:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760730.013760:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.013771:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.013910:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760730.013954:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493702656:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760730.057505:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5635, 5635] 00000080:00200000:1.0:1789760730.057532:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760730.057538:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760730.057554:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23085056 i_blocks 47111 00000020:00200000:1.0:1789760730.057558:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760730.057569:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760730.057613:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23085056, count: 4096 00000080:00200000:1.0:1789760730.057621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23085056, bytes: 4096 00000080:00200000:1.0:1789760730.057633:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760730.057641:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760730.057643:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23085056, 23089152) 0 0 00000080:00200000:1.0:1789760730.057648:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760730.057667:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5636, 5636] 00020000:00200000:1.0:1789760730.057675:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23085056 chunk: [23085056, 23089152) 23089152, 4096 00020000:00200000:1.0:1789760730.057680:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760730.057700:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5636, 5636] 00000008:00010000:1.0:1789760730.057719:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760730.057726:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760730.057738:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23085056, 23089152) append 0 append lockpos 0 00000080:00200000:1.0:1789760730.057753:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23085056=0x1604000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760730.057809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760730.057812:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23089152 00000020:00010000:1.0:1789760730.057821:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760730.057824:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.057828:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.057832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760730.057834:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760730.057845:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760730.057852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760730.057868:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760730.057966:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 23085056<>23089152 00000100:00100000:1.0:1789760730.058009:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760730.058026:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493702912:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.101118:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972955 gen 3 00000008:00010000:1.0:1789760730.101182:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760730.101192:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.101203:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.101378:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760730.101462:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493702912:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.102573:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493703040:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760730.136177:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760730.136191:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.136201:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.136309:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760730.136341:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493703040:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760730.138026:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5637, 5637] 00000080:00200000:3.0:1789760730.138049:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.138056:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.138075:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23093248 i_blocks 47111 00000020:00200000:3.0:1789760730.138080:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.138098:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.138131:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23093248, count: 4096 00000080:00200000:3.0:1789760730.138139:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23093248, bytes: 4096 00000080:00200000:3.0:1789760730.138153:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.138162:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.138164:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23093248, 23097344) 0 0 00000080:00200000:3.0:1789760730.138168:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.138185:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5638, 5638] 00020000:00200000:3.0:1789760730.138194:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23093248 chunk: [23093248, 23097344) 23097344, 4096 00020000:00200000:3.0:1789760730.138198:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.138526:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5638, 5638] 00000008:00010000:3.0:1789760730.138548:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.138553:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760730.138566:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23093248, 23097344) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.138579:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23093248=0x1606000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.138642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.138645:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23097344 00000020:00010000:3.0:1789760730.138652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.138656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.138659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.138662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.138665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.138673:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.138679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.139121:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760730.139779:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 23093248<>23097344 00000100:00100000:2.0:1789760730.139847:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760730.141690:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493703168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760730.174487:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972957 gen 3 00000008:00010000:3.0:1789760730.174559:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:3.0:1789760730.174567:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.174577:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.174598:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:3.0:1789760730.174630:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493703168:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.174761:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5638, 5638] 00000080:00200000:2.0:1789760730.174777:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.174783:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.174800:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23097344 i_blocks 47111 00000020:00200000:2.0:1789760730.174804:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.174814:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.174846:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23097344, count: 4096 00000080:00200000:2.0:1789760730.174853:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23097344, bytes: 4096 00000080:00200000:2.0:1789760730.174861:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.174870:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.174873:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23097344, 23101440) 0 0 00000080:00200000:2.0:1789760730.174877:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.174895:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5639, 5639] 00020000:00200000:2.0:1789760730.174903:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23097344 chunk: [23097344, 23101440) 23101440, 4096 00020000:00200000:2.0:1789760730.174908:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.174930:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5639, 5639] 00000008:00010000:2.0:1789760730.174949:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.174956:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.174967:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23097344, 23101440) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.174982:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23097344=0x1607000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.175041:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.175044:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23101440 00000020:00010000:2.0:1789760730.175052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.175056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.175059:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.175063:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.175066:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.175074:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.175081:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.175096:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760730.175196:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23097344<>23101440 00000100:00100000:3.0:1789760730.175535:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760730.175555:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703296:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760730.231388:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972958 gen 3 00000008:00010000:3.0:1789760730.231612:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760730.231621:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.231636:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.231783:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5639, 5639] 00000008:00010000:3.0:1789760730.231786:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000080:00200000:2.0:1789760730.231801:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.231807:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.231823:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23101440 i_blocks 47111 00000020:00200000:2.0:1789760730.231826:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.231838:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.231874:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23101440, count: 4096 00000080:00200000:2.0:1789760730.231882:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23101440, bytes: 4096 00000100:00100000:3.0:1789760730.232032:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.232275:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.232286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.232289:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23101440, 23105536) 0 0 00000080:00200000:2.0:1789760730.232293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.232310:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5640, 5640] 00020000:00200000:2.0:1789760730.232319:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23101440 chunk: [23101440, 23105536) 23105536, 4096 00020000:00200000:2.0:1789760730.232324:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.232346:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5640, 5640] 00000008:00010000:2.0:1789760730.232362:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.232367:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.232377:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23101440, 23105536) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.232389:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23101440=0x1608000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.232557:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.232560:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23105536 00000020:00010000:2.0:1789760730.232570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.232573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.232577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.232581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.232584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.232592:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.232599:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.232617:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760730.232725:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23101440<>23105536 00000100:00100000:3.0:1789760730.232796:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703424:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760730.278321:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760730.278337:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.278350:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.278373:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760730.278452:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.278701:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5640, 5640] 00000080:00200000:2.0:1789760730.278718:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.278724:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.278740:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23105536 i_blocks 47111 00000020:00200000:2.0:1789760730.278744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.278755:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.278789:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23105536, count: 4096 00000080:00200000:2.0:1789760730.278797:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23105536, bytes: 4096 00000080:00200000:2.0:1789760730.278807:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.278827:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.278829:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23105536, 23109632) 0 0 00000080:00200000:2.0:1789760730.278834:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.278854:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5641, 5641] 00020000:00200000:2.0:1789760730.278861:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23105536 chunk: [23105536, 23109632) 23109632, 4096 00020000:00200000:2.0:1789760730.278866:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.278885:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5641, 5641] 00000008:00010000:2.0:1789760730.279032:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.279039:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.279052:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23105536, 23109632) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.279065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23105536=0x1609000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.279122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.279125:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23109632 00000020:00010000:2.0:1789760730.279207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.279211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.279214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.279218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.279221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.279230:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.279238:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.279254:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760730.279365:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23105536<>23109632 00000100:00100000:3.0:1789760730.279573:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760730.279596:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703552:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760730.313681:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972960 gen 3 00000008:00010000:3.0:1789760730.313746:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760730.313754:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.313765:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.313787:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760730.313826:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703552:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.314002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5641, 5641] 00000080:00200000:2.0:1789760730.314018:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.314024:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.314040:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23109632 i_blocks 47111 00000020:00200000:2.0:1789760730.314044:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.314055:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.314087:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23109632, count: 4096 00000080:00200000:2.0:1789760730.314095:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23109632, bytes: 4096 00000080:00200000:2.0:1789760730.314105:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.314114:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.314117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23109632, 23113728) 0 0 00000080:00200000:2.0:1789760730.314122:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.314138:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5642, 5642] 00020000:00200000:2.0:1789760730.314146:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23109632 chunk: [23109632, 23113728) 23113728, 4096 00020000:00200000:2.0:1789760730.314151:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.314170:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5642, 5642] 00000008:00010000:2.0:1789760730.314189:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.314195:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.314206:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23109632, 23113728) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.314221:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23109632=0x160a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.314279:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.314282:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23113728 00000020:00010000:2.0:1789760730.314291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.314295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.314298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.314302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.314305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.314313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.314319:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.314333:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760730.314486:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23109632<>23113728 00000100:00100000:3.0:1789760730.314772:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703680:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760730.348737:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760730.356108:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972961 gen 3 00000008:00010000:3.0:1789760730.356161:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760730.356169:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.356183:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.356242:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760730.356272:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493703680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.356353:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5642, 5642] 00000080:00200000:2.0:1789760730.356370:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.356377:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.356411:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23113728 i_blocks 47111 00000020:00200000:2.0:1789760730.356428:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.356439:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.356544:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23113728, count: 4096 00000080:00200000:2.0:1789760730.356553:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23113728, bytes: 4096 00000080:00200000:2.0:1789760730.356564:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.356574:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.356577:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23113728, 23117824) 0 0 00000080:00200000:2.0:1789760730.356582:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.356602:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5643, 5643] 00020000:00200000:2.0:1789760730.356612:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23113728 chunk: [23113728, 23117824) 23117824, 4096 00020000:00200000:2.0:1789760730.356616:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.356639:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5643, 5643] 00000008:00010000:2.0:1789760730.356658:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.356665:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.356678:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23113728, 23117824) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.356692:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23113728=0x160b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.356750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.356753:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23117824 00000020:00010000:2.0:1789760730.356763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.356766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.356769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.356774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.356777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.356785:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.356792:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.356809:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760730.357096:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23113728<>23117824 00000100:00100000:3.0:1789760730.357164:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493703808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760730.400493:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972962 gen 3 00000008:00010000:3.0:1789760730.400547:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760730.400555:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.400567:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.400645:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760730.400676:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493703808:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760730.402193:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5643, 5643] 00000080:00200000:1.0:1789760730.402215:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760730.402222:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760730.402239:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23117824 i_blocks 47111 00000020:00200000:1.0:1789760730.402244:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760730.402260:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760730.402297:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23117824, count: 4096 00000080:00200000:1.0:1789760730.402305:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23117824, bytes: 4096 00000080:00200000:1.0:1789760730.402320:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760730.402329:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760730.402331:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23117824, 23121920) 0 0 00000080:00200000:1.0:1789760730.402337:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760730.402357:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5644, 5644] 00020000:00200000:1.0:1789760730.402366:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23117824 chunk: [23117824, 23121920) 23121920, 4096 00020000:00200000:1.0:1789760730.402371:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760730.402410:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5644, 5644] 00000008:00010000:1.0:1789760730.402440:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760730.402446:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760730.402460:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23117824, 23121920) append 0 append lockpos 0 00000080:00200000:1.0:1789760730.402473:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23117824=0x160c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760730.402530:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760730.402533:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23121920 00000020:00010000:1.0:1789760730.402541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760730.402544:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.402548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.402551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760730.402554:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760730.402565:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760730.402572:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760730.402589:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760730.402694:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 23117824<>23121920 00000100:00100000:1.0:1789760730.402739:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760730.402764:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493703936:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.455510:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972963 gen 3 00000008:00010000:1.0:1789760730.455562:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760730.455571:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.455581:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.455608:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760730.455636:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493703936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760730.523976:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5645, 5645] 00000080:00200000:2.0:1789760730.523993:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760730.523999:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760730.524015:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23126016 i_blocks 47111 00000020:00200000:2.0:1789760730.524019:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760730.524029:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760730.524066:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23126016, count: 4096 00000080:00200000:2.0:1789760730.524074:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23126016, bytes: 4096 00000080:00200000:2.0:1789760730.524083:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760730.524092:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760730.524094:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23126016, 23130112) 0 0 00000080:00200000:2.0:1789760730.524099:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760730.524117:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5646, 5646] 00020000:00200000:2.0:1789760730.524125:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23126016 chunk: [23126016, 23130112) 23130112, 4096 00020000:00200000:2.0:1789760730.524130:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760730.524150:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5646, 5646] 00000008:00010000:2.0:1789760730.524167:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760730.524173:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760730.524184:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23126016, 23130112) append 0 append lockpos 0 00000080:00200000:2.0:1789760730.524198:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23126016=0x160e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760730.524255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760730.524258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23130112 00000020:00010000:2.0:1789760730.524267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760730.524270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.524353:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760730.524360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760730.524363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760730.524373:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760730.524380:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760730.524436:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760730.524773:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23126016<>23130112 00000100:00100000:3.0:1789760730.524825:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760730.528639:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493704192:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760730.569877:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760730.569896:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760730.569909:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760730.569937:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760730.569969:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493704192:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.578559:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493704320:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.626194:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972966 gen 3 00000008:00010000:1.0:1789760730.626262:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760730.626271:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.626282:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.626307:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760730.626338:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493704320:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.627103:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493704448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.690681:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972967 gen 3 00000008:00010000:1.0:1789760730.690740:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760730.690748:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.690759:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.690784:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760730.690822:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493704448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760730.691846:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760730.691958:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493704576:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760730.743503:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760730.743522:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.743534:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.743559:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760730.743592:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493704576:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760730.744105:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5649, 5649] 00000080:00200000:3.0:1789760730.744121:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.744128:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.744143:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23142400 i_blocks 47111 00000020:00200000:3.0:1789760730.744147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.744158:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.744190:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23142400, count: 4096 00000080:00200000:3.0:1789760730.744197:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23142400, bytes: 4096 00000080:00200000:3.0:1789760730.744210:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.744219:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.744222:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23142400, 23146496) 0 0 00000080:00200000:3.0:1789760730.744226:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.744244:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5650, 5650] 00020000:00200000:3.0:1789760730.744252:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23142400 chunk: [23142400, 23146496) 23146496, 4096 00020000:00200000:3.0:1789760730.744257:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.744278:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5650, 5650] 00000008:00010000:3.0:1789760730.744296:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.744302:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760730.744314:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23142400, 23146496) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.744329:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23142400=0x1612000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.744389:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.744974:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23146496 00000020:00010000:3.0:1789760730.744988:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.744991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.744995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.744999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.745002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.745012:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.745020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.745044:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760730.745155:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 23142400<>23146496 00000100:00100000:2.0:1789760730.745243:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493704704:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760730.809238:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972969 gen 3 00000008:00010000:2.0:1789760730.809313:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760730.809321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.809331:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.809353:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760730.809384:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493704704:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760730.809796:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5650, 5650] 00000080:00200000:3.0:1789760730.809813:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.809871:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.809888:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23146496 i_blocks 47111 00000020:00200000:3.0:1789760730.809963:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.809975:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.810012:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23146496, count: 4096 00000080:00200000:3.0:1789760730.810020:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23146496, bytes: 4096 00000080:00200000:3.0:1789760730.810034:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.810044:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.810046:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23146496, 23150592) 0 0 00000080:00200000:3.0:1789760730.810051:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.810066:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5651, 5651] 00020000:00200000:3.0:1789760730.810075:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23146496 chunk: [23146496, 23150592) 23150592, 4096 00020000:00200000:3.0:1789760730.810079:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.810098:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5651, 5651] 00000008:00010000:3.0:1789760730.810114:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.810120:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760730.810132:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23146496, 23150592) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.810145:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23146496=0x1613000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.810207:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.810210:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23150592 00000020:00010000:3.0:1789760730.810217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.810221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.810224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.810227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.810230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.810238:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.810245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.810263:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760730.810371:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 23146496<>23150592 00000100:00100000:2.0:1789760730.810762:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493704832:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760730.881137:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972970 gen 3 00000008:00010000:2.0:1789760730.881189:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760730.881197:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.881209:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.881229:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760730.881255:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493704832:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760730.881629:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5651, 5651] 00000080:00200000:3.0:1789760730.881645:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.881651:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.881667:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23150592 i_blocks 47111 00000020:00200000:3.0:1789760730.881670:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.881680:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.881713:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23150592, count: 4096 00000080:00200000:3.0:1789760730.881720:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23150592, bytes: 4096 00000080:00200000:3.0:1789760730.881729:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.881737:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.881740:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23150592, 23154688) 0 0 00000080:00200000:3.0:1789760730.881744:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.881936:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5652, 5652] 00020000:00200000:3.0:1789760730.881947:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23150592 chunk: [23150592, 23154688) 23154688, 4096 00020000:00200000:3.0:1789760730.881952:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.881972:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5652, 5652] 00000008:00010000:3.0:1789760730.882151:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.882159:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760730.882173:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23150592, 23154688) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.882187:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23150592=0x1614000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.882252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.882255:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23154688 00000020:00010000:3.0:1789760730.882264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.882267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.882270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.882273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.882276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.882284:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.882292:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.882310:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760730.882697:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 23150592<>23154688 00000100:00100000:3.0:1789760730.882742:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760730.882775:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493704960:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760730.926586:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760730.926601:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.926612:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.926640:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760730.926669:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493704960:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760730.927574:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5652, 5652] 00000080:00200000:1.0:1789760730.927595:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760730.927602:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760730.927619:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23154688 i_blocks 47111 00000020:00200000:1.0:1789760730.927624:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760730.927635:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760730.927674:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23154688, count: 4096 00000080:00200000:1.0:1789760730.927682:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23154688, bytes: 4096 00000080:00200000:1.0:1789760730.927693:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760730.927703:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760730.927705:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23154688, 23158784) 0 0 00000080:00200000:1.0:1789760730.927710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760730.927726:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5653, 5653] 00020000:00200000:1.0:1789760730.927735:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23154688 chunk: [23154688, 23158784) 23158784, 4096 00020000:00200000:1.0:1789760730.927739:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760730.927759:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5653, 5653] 00000008:00010000:1.0:1789760730.927777:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760730.927784:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760730.927973:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23154688, 23158784) append 0 append lockpos 0 00000080:00200000:1.0:1789760730.927998:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23154688=0x1615000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760730.928072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760730.928075:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23158784 00000020:00010000:1.0:1789760730.928084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760730.928087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.928090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.928094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760730.928097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760730.928109:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760730.928117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760730.928136:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760730.928252:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 23154688<>23158784 00000080:00200000:1.0:1789760731.003814:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5653, 5653] 00000080:00200000:1.0:1789760731.003838:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760731.003844:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760731.003861:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23158784 i_blocks 47111 00000020:00200000:1.0:1789760731.003865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760731.003874:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760731.003908:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23158784, count: 4096 00000080:00200000:1.0:1789760731.003916:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23158784, bytes: 4096 00000080:00200000:1.0:1789760731.003928:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760731.003936:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760731.003938:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23158784, 23162880) 0 0 00000080:00200000:1.0:1789760731.003943:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760731.003959:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5654, 5654] 00020000:00200000:1.0:1789760731.003968:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23158784 chunk: [23158784, 23162880) 23162880, 4096 00020000:00200000:1.0:1789760731.003972:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760731.004134:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5654, 5654] 00000008:00010000:1.0:1789760731.004154:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760731.004159:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760731.004171:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23158784, 23162880) append 0 append lockpos 0 00000080:00200000:1.0:1789760731.004187:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23158784=0x1616000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760731.004246:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760731.004249:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23162880 00000020:00010000:1.0:1789760731.004257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760731.004261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.004264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.004268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760731.004272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760731.004283:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760731.004291:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760731.004307:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760731.004703:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 23158784<>23162880 00000080:00200000:1.0:1789760731.059285:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5654, 5654] 00000080:00200000:1.0:1789760731.059302:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760731.059309:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760731.059325:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23162880 i_blocks 47111 00000020:00200000:1.0:1789760731.059329:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760731.059340:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760731.059375:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23162880, count: 4096 00000080:00200000:1.0:1789760731.059382:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23162880, bytes: 4096 00000080:00200000:1.0:1789760731.059436:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760731.059446:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760731.059448:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23162880, 23166976) 0 0 00000080:00200000:1.0:1789760731.059452:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760731.059468:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5655, 5655] 00020000:00200000:1.0:1789760731.059476:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23162880 chunk: [23162880, 23166976) 23166976, 4096 00020000:00200000:1.0:1789760731.059480:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760731.059499:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5655, 5655] 00000008:00010000:1.0:1789760731.059517:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760731.059523:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760731.059536:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23162880, 23166976) append 0 append lockpos 0 00000080:00200000:1.0:1789760731.059551:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23162880=0x1617000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760731.059608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760731.059610:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23166976 00000020:00010000:1.0:1789760731.059619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760731.059623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.059626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.059630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760731.059633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760731.059645:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760731.059652:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760731.059668:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760731.059774:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 23162880<>23166976 00000080:00200000:1.0:1789760731.105751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5655, 5655] 00000080:00200000:1.0:1789760731.105767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760731.105773:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760731.105789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23166976 i_blocks 47111 00000020:00200000:1.0:1789760731.105793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760731.105804:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760731.105840:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23166976, count: 4096 00000080:00200000:1.0:1789760731.105848:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23166976, bytes: 4096 00000080:00200000:1.0:1789760731.105860:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760731.105870:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760731.105872:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23166976, 23171072) 0 0 00000080:00200000:1.0:1789760731.105876:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760731.106027:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5656, 5656] 00020000:00200000:1.0:1789760731.106038:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23166976 chunk: [23166976, 23171072) 23171072, 4096 00020000:00200000:1.0:1789760731.106042:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760731.106064:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5656, 5656] 00000008:00010000:1.0:1789760731.106083:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760731.106089:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760731.106102:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23166976, 23171072) append 0 append lockpos 0 00000080:00200000:1.0:1789760731.106117:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23166976=0x1618000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760731.106180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760731.106182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23171072 00000020:00010000:1.0:1789760731.106191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760731.106194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.106198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.106202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760731.106205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760731.106216:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760731.106223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760731.106239:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760731.106344:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 23166976<>23171072 00000080:00200000:1.0:1789760731.149616:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5656, 5656] 00000080:00200000:1.0:1789760731.149634:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760731.149640:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760731.149658:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23171072 i_blocks 47111 00000020:00200000:1.0:1789760731.149662:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760731.149673:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760731.149708:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23171072, count: 4096 00000080:00200000:1.0:1789760731.149726:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23171072, bytes: 4096 00000080:00200000:1.0:1789760731.149740:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760731.149748:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760731.149751:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23171072, 23175168) 0 0 00000080:00200000:1.0:1789760731.149755:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760731.149771:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5657, 5657] 00020000:00200000:1.0:1789760731.149779:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23171072 chunk: [23171072, 23175168) 23175168, 4096 00020000:00200000:1.0:1789760731.149783:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760731.149802:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5657, 5657] 00000008:00010000:1.0:1789760731.149819:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760731.149825:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760731.149837:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23171072, 23175168) append 0 append lockpos 0 00000080:00200000:1.0:1789760731.149851:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23171072=0x1619000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760731.149920:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760731.149923:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23175168 00000020:00010000:1.0:1789760731.149931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760731.149935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.149938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.149942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760731.149945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760731.149956:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760731.149974:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760731.149990:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760731.150112:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 23171072<>23175168 00000080:00200000:1.0:1789760731.224788:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5657, 5657] 00000080:00200000:1.0:1789760731.224806:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760731.224812:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760731.224828:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23175168 i_blocks 47111 00000020:00200000:1.0:1789760731.224832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760731.224843:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760731.224878:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23175168, count: 4096 00000080:00200000:1.0:1789760731.224886:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23175168, bytes: 4096 00000080:00200000:1.0:1789760731.224899:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760731.225021:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760731.225025:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23175168, 23179264) 0 0 00000080:00200000:1.0:1789760731.225030:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760731.225047:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5658, 5658] 00020000:00200000:1.0:1789760731.225056:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23175168 chunk: [23175168, 23179264) 23179264, 4096 00020000:00200000:1.0:1789760731.225061:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760731.225083:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5658, 5658] 00000008:00010000:1.0:1789760731.225103:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760731.225109:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760731.225122:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23175168, 23179264) append 0 append lockpos 0 00000080:00200000:1.0:1789760731.225137:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23175168=0x161a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760731.225198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760731.225200:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23179264 00000020:00010000:1.0:1789760731.225209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760731.225212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.225216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760731.225220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760731.225224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760731.225235:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760731.225243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760731.225260:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760731.225368:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 23175168<>23179264 00000080:00200000:2.0:1789760731.279303:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5658, 5658] 00000080:00200000:2.0:1789760731.279320:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760731.279327:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760731.279342:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23179264 i_blocks 47111 00000020:00200000:2.0:1789760731.279346:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760731.279356:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760731.279391:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23179264, count: 4096 00000080:00200000:2.0:1789760731.279441:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23179264, bytes: 4096 00000080:00200000:2.0:1789760731.279452:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760731.279461:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760731.279463:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23179264, 23183360) 0 0 00000080:00200000:2.0:1789760731.279468:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760731.279624:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5659, 5659] 00020000:00200000:2.0:1789760731.279633:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23179264 chunk: [23179264, 23183360) 23183360, 4096 00020000:00200000:2.0:1789760731.279638:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760731.279660:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5659, 5659] 00000008:00010000:2.0:1789760731.279680:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760731.279686:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760731.279699:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23179264, 23183360) append 0 append lockpos 0 00000080:00200000:2.0:1789760731.279713:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23179264=0x161b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760731.279772:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760731.279775:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23183360 00000020:00010000:2.0:1789760731.279782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760731.279786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760731.279789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760731.279792:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760731.279796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760731.279804:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760731.279812:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760731.280365:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760731.280524:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23179264<>23183360 00000100:00100000:3.0:1789760731.280609:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493705856:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760731.322541:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:2.0:1789760731.322557:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.322567:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.322594:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:2.0:1789760731.322622:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493705856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760731.379312:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5660, 5660] 00000080:00200000:2.0:1789760731.379328:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760731.379334:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760731.379350:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23187456 i_blocks 47111 00000020:00200000:2.0:1789760731.379354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760731.379364:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760731.379452:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23187456, count: 4096 00000080:00200000:2.0:1789760731.379461:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23187456, bytes: 4096 00000080:00200000:2.0:1789760731.379476:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760731.379485:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760731.379487:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23187456, 23191552) 0 0 00000080:00200000:2.0:1789760731.379492:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760731.379508:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5661, 5661] 00020000:00200000:2.0:1789760731.379701:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23187456 chunk: [23187456, 23191552) 23191552, 4096 00020000:00200000:2.0:1789760731.379706:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760731.379728:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5661, 5661] 00000008:00010000:2.0:1789760731.379746:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760731.379752:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760731.379765:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23187456, 23191552) append 0 append lockpos 0 00000080:00200000:2.0:1789760731.379779:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23187456=0x161d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760731.379841:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760731.379843:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23191552 00000020:00010000:2.0:1789760731.379851:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760731.379855:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760731.379858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760731.379862:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760731.379866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760731.379874:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760731.379881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760731.379898:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760731.380001:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23187456<>23191552 00000100:00100000:3.0:1789760731.380083:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493706112:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760731.428602:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972980 gen 3 00000008:00010000:2.0:1789760731.428657:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:2.0:1789760731.428666:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.428679:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.428764:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:2.0:1789760731.428795:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493706112:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.429747:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493706240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.472641:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972981 gen 3 00000008:00010000:1.0:1789760731.472694:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760731.472703:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.472714:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.472735:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760731.472764:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493706240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.473572:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.510794:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972982 gen 3 00000008:00010000:1.0:1789760731.511565:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760731.511577:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.511590:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.511616:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760731.511652:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.513264:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760731.513290:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.562200:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972983 gen 3 00000008:00010000:1.0:1789760731.562253:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760731.562262:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.562273:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.562296:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760731.562325:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.563037:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706624:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.594216:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972984 gen 3 00000008:00010000:1.0:1789760731.594264:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760731.594272:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.594282:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.594304:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760731.594332:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706624:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.595467:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760731.595488:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706752:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760731.638491:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760731.638508:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.638520:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.638542:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760731.638572:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706752:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.639230:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.679208:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972986 gen 3 00000008:00010000:1.0:1789760731.679277:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760731.679286:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.679297:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.679321:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760731.679351:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493706880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.680068:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760731.680087:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707008:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760731.693064:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493707264:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760731.693341:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760731.693350:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493707136:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760731.693439:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493707392:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760731.697075:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493707136:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760731.697104:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493707392:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760731.697116:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493707264:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760731.723453:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972987 gen 3 00000008:00010000:1.0:1789760731.723506:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760731.723515:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.723973:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.724342:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760731.724579:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707008:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.725211:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707520:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760731.761795:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760731.761811:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.761823:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.761904:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760731.761936:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707520:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.762864:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493707648:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.801153:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972989 gen 3 00000008:00010000:1.0:1789760731.801235:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760731.801245:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.801255:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.801277:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760731.801313:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493707648:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760731.802187:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5670, 5670] 00000080:00200000:3.0:1789760731.802207:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.802214:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.802231:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23228416 i_blocks 47111 00000020:00200000:3.0:1789760731.802235:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.802247:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.802281:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23228416, count: 4096 00000080:00200000:3.0:1789760731.802289:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23228416, bytes: 4096 00000080:00200000:3.0:1789760731.802303:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.802313:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.802316:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23228416, 23232512) 0 0 00000080:00200000:3.0:1789760731.802320:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.802339:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5671, 5671] 00020000:00200000:3.0:1789760731.802349:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23228416 chunk: [23228416, 23232512) 23232512, 4096 00020000:00200000:3.0:1789760731.802353:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.802374:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5671, 5671] 00000008:00010000:3.0:1789760731.802605:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.802615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760731.802630:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23228416, 23232512) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.802645:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23228416=0x1627000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.802715:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.802718:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23232512 00000020:00010000:3.0:1789760731.802726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.802729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.802733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.802736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.802739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.802747:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.802754:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.802772:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760731.802955:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 23228416<>23232512 00000100:00100000:2.0:1789760731.803115:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493707776:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760731.849224:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972990 gen 3 00000008:00010000:3.0:1789760731.849272:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760731.849279:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760731.849290:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760731.849314:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760731.849341:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493707776:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.852383:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.898928:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972991 gen 3 00000008:00010000:1.0:1789760731.899592:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760731.899600:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.899611:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.899694:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760731.899724:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493707904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.900826:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493708032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.956015:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972992 gen 3 00000008:00010000:1.0:1789760731.956066:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760731.956075:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760731.956086:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760731.956173:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760731.956202:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493708032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760731.957115:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493708160:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760732.007244:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760732.007261:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.007272:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.007328:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760732.007357:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493708160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.007569:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5674, 5674] 00000080:00200000:2.0:1789760732.007585:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.007591:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.007607:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23244800 i_blocks 47111 00000020:00200000:2.0:1789760732.007611:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.007907:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.007943:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23244800, count: 4096 00000080:00200000:2.0:1789760732.007952:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23244800, bytes: 4096 00000080:00200000:2.0:1789760732.007964:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.007973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.007976:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23244800, 23248896) 0 0 00000080:00200000:2.0:1789760732.007980:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.008529:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5675, 5675] 00020000:00200000:2.0:1789760732.008541:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23244800 chunk: [23244800, 23248896) 23248896, 4096 00020000:00200000:2.0:1789760732.008546:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.008571:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5675, 5675] 00000008:00010000:2.0:1789760732.008590:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.008596:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.008609:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23244800, 23248896) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.008625:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23244800=0x162b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.008686:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.008688:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23248896 00000020:00010000:2.0:1789760732.008696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.008700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.008703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.008706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.008709:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.008717:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.008724:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.008742:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760732.008930:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23244800<>23248896 00000100:00100000:3.0:1789760732.009037:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.043244:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972994 gen 3 00000008:00010000:3.0:1789760732.043314:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760732.043323:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.043335:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.043357:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760732.043441:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.043925:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5675, 5675] 00000080:00200000:2.0:1789760732.043944:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.043950:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.043967:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23248896 i_blocks 47111 00000020:00200000:2.0:1789760732.043971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.044215:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.044253:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23248896, count: 4096 00000080:00200000:2.0:1789760732.044264:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23248896, bytes: 4096 00000080:00200000:2.0:1789760732.044277:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.044286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.044289:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23248896, 23252992) 0 0 00000080:00200000:2.0:1789760732.044294:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.044312:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5676, 5676] 00020000:00200000:2.0:1789760732.044321:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23248896 chunk: [23248896, 23252992) 23252992, 4096 00020000:00200000:2.0:1789760732.044326:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.044348:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5676, 5676] 00000008:00010000:2.0:1789760732.044367:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.044373:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.044384:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23248896, 23252992) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.044437:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23248896=0x162c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.046138:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.046147:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23252992 00000020:00010000:2.0:1789760732.046162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.046166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.046171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.046175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.046177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.046188:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.046197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.046220:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760732.046336:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23248896<>23252992 00000100:00100000:3.0:1789760732.047025:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493708416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.111860:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972995 gen 3 00000008:00010000:3.0:1789760732.111909:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760732.111917:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.111929:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.111948:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760732.111980:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493708416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.112127:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5676, 5676] 00000080:00200000:2.0:1789760732.112143:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.112148:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.112164:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23252992 i_blocks 47111 00000020:00200000:2.0:1789760732.112167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.112177:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.112209:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23252992, count: 4096 00000080:00200000:2.0:1789760732.112216:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23252992, bytes: 4096 00000080:00200000:2.0:1789760732.112224:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.112233:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.112235:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23252992, 23257088) 0 0 00000080:00200000:2.0:1789760732.112240:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.112257:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5677, 5677] 00020000:00200000:2.0:1789760732.112264:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23252992 chunk: [23252992, 23257088) 23257088, 4096 00020000:00200000:2.0:1789760732.112269:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.112287:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5677, 5677] 00000008:00010000:2.0:1789760732.112304:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.112310:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.112321:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23252992, 23257088) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.112334:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23252992=0x162d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.112386:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.112388:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23257088 00000020:00010000:2.0:1789760732.112443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.112448:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.112451:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.112454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.112457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.112465:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.112472:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.112485:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760732.112602:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23252992<>23257088 00000100:00100000:3.0:1789760732.112809:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708544:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.151920:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972996 gen 3 00000008:00010000:3.0:1789760732.151975:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760732.151984:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.151996:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.152017:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760732.152048:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708544:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.152208:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5677, 5677] 00000080:00200000:2.0:1789760732.152225:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.152231:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.152248:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23257088 i_blocks 47111 00000020:00200000:2.0:1789760732.152252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.152264:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.152302:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23257088, count: 4096 00000080:00200000:2.0:1789760732.152309:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23257088, bytes: 4096 00000080:00200000:2.0:1789760732.152321:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.152331:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.152333:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23257088, 23261184) 0 0 00000080:00200000:2.0:1789760732.152338:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.152355:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5678, 5678] 00020000:00200000:2.0:1789760732.152364:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23257088 chunk: [23257088, 23261184) 23261184, 4096 00020000:00200000:2.0:1789760732.152370:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.152391:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5678, 5678] 00000008:00010000:2.0:1789760732.152453:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.152459:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.152472:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23257088, 23261184) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.152487:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23257088=0x162e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.152550:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.152552:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23261184 00000020:00010000:2.0:1789760732.152562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.152566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.152568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.152571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.152574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.152581:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.152588:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.152605:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760732.152713:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23257088<>23261184 00000100:00100000:3.0:1789760732.153386:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760732.153469:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.183713:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972997 gen 3 00000008:00010000:3.0:1789760732.183770:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760732.183780:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.183793:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.183825:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760732.183859:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708672:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.184118:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5678, 5678] 00000080:00200000:2.0:1789760732.184135:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.184141:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.184157:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23261184 i_blocks 47111 00000020:00200000:2.0:1789760732.184161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.184172:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.184207:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23261184, count: 4096 00000080:00200000:2.0:1789760732.184215:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23261184, bytes: 4096 00000080:00200000:2.0:1789760732.184225:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.184234:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.184236:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23261184, 23265280) 0 0 00000080:00200000:2.0:1789760732.184241:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.184259:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5679, 5679] 00020000:00200000:2.0:1789760732.184268:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23261184 chunk: [23261184, 23265280) 23265280, 4096 00020000:00200000:2.0:1789760732.184273:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.184293:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5679, 5679] 00000008:00010000:2.0:1789760732.184311:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.184316:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.184327:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23261184, 23265280) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.184340:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23261184=0x162f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.184448:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.184548:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23265280 00000020:00010000:2.0:1789760732.184558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.184561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.184564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.184568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.184570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.184579:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.184586:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.184602:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760732.184704:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23261184<>23265280 00000100:00100000:3.0:1789760732.184837:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760732.184856:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493708800:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760732.222558:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760732.222576:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.222588:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.222609:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760732.222644:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493708800:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.222978:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5679, 5679] 00000080:00200000:2.0:1789760732.222996:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.223002:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.223019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23265280 i_blocks 47111 00000020:00200000:2.0:1789760732.223022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.223034:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.223069:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23265280, count: 4096 00000080:00200000:2.0:1789760732.223077:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23265280, bytes: 4096 00000080:00200000:2.0:1789760732.223087:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.223097:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.223099:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23265280, 23269376) 0 0 00000080:00200000:2.0:1789760732.223104:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.223119:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5680, 5680] 00020000:00200000:2.0:1789760732.223128:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23265280 chunk: [23265280, 23269376) 23269376, 4096 00020000:00200000:2.0:1789760732.223133:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.223151:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5680, 5680] 00000008:00010000:2.0:1789760732.223170:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.223176:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.223187:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23265280, 23269376) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.223200:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23265280=0x1630000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.223261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.223263:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23269376 00000020:00010000:2.0:1789760732.223272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.223276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.223279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.223283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.223286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.223294:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.223301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.223317:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760732.223484:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23265280<>23269376 00000100:00100000:3.0:1789760732.223592:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760732.223614:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708928:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.276463:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972998 gen 3 00000100:00100000:3.0:1789760732.276504:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80a80 x1876699493708928/t4294972999(4294972999) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760732.276552:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760732.276559:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.276568:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.276587:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760732.276609:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493708928:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.276692:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5680, 5680] 00000080:00200000:2.0:1789760732.276708:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.276713:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.276727:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23269376 i_blocks 47111 00000020:00200000:2.0:1789760732.276730:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.276740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.276774:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23269376, count: 4096 00000080:00200000:2.0:1789760732.276781:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23269376, bytes: 4096 00000080:00200000:2.0:1789760732.276799:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.276807:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.276810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23269376, 23273472) 0 0 00000080:00200000:2.0:1789760732.276814:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.276835:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5681, 5681] 00020000:00200000:2.0:1789760732.276843:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23269376 chunk: [23269376, 23273472) 23273472, 4096 00020000:00200000:2.0:1789760732.276848:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.276866:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5681, 5681] 00000008:00010000:2.0:1789760732.277026:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.277033:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.277046:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23269376, 23273472) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.277059:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23269376=0x1631000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.277112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.277115:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23273472 00000020:00010000:2.0:1789760732.277124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.277127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.277130:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.277134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.277137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.277144:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.277151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.277166:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760732.277265:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23269376<>23273472 00000100:00100000:3.0:1789760732.277350:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760732.277366:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493709056:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.319324:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294972999 gen 3 00000100:00100000:3.0:1789760732.319367:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81500 x1876699493709056/t4294973000(4294973000) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760732.319469:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760732.319477:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.319488:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.319506:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760732.319530:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493709056:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.320338:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493709184:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.358834:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973001 gen 3 00000008:00010000:1.0:1789760732.358914:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760732.358923:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.358933:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.358946:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760732.358977:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493709184:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.360277:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493709312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.393390:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973002 gen 3 00000008:00010000:1.0:1789760732.393479:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760732.393489:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.393501:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.393579:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760732.393609:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493709312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.394172:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493709440:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.450145:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973003 gen 3 00000008:00010000:1.0:1789760732.450197:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760732.450206:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.450216:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.450237:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760732.450265:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493709440:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760732.455073:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:1.0:1789760732.455209:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 23285760<>23289856 00000080:00200000:2.0:1789760732.542723:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5685, 5685] 00000080:00200000:2.0:1789760732.542740:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.542745:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.542759:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23289856 i_blocks 47111 00000020:00200000:2.0:1789760732.542763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.542774:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.542809:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23289856, count: 4096 00000080:00200000:2.0:1789760732.542826:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23289856, bytes: 4096 00000080:00200000:2.0:1789760732.542837:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.542846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.542848:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23289856, 23293952) 0 0 00000080:00200000:2.0:1789760732.542853:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.542868:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5686, 5686] 00020000:00200000:2.0:1789760732.543006:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23289856 chunk: [23289856, 23293952) 23293952, 4096 00020000:00200000:2.0:1789760732.543012:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.543033:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5686, 5686] 00000008:00010000:2.0:1789760732.543053:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.543059:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.543071:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23289856, 23293952) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.543084:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23289856=0x1636000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.543139:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.543142:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23293952 00000020:00010000:2.0:1789760732.543150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.543153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.543156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.543159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.543162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.543169:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.543176:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.543190:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760732.543303:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23289856<>23293952 00000100:00100000:3.0:1789760732.543382:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493709696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.576316:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973005 gen 3 00000008:00010000:3.0:1789760732.576383:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760732.576391:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.576929:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.577006:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760732.577037:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493709696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.581853:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5686, 5686] 00000080:00200000:2.0:1789760732.581884:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.581891:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.581911:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23293952 i_blocks 47111 00000020:00200000:2.0:1789760732.581917:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.581936:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.581970:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23293952, count: 4096 00000080:00200000:2.0:1789760732.581980:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23293952, bytes: 4096 00000080:00200000:2.0:1789760732.581996:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.582006:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.582008:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23293952, 23298048) 0 0 00000080:00200000:2.0:1789760732.582013:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.582030:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5687, 5687] 00020000:00200000:2.0:1789760732.582038:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23293952 chunk: [23293952, 23298048) 23298048, 4096 00020000:00200000:2.0:1789760732.582043:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.582065:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5687, 5687] 00000008:00010000:2.0:1789760732.582356:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.582363:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.582377:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23293952, 23298048) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.582428:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23293952=0x1637000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.583276:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.583281:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23298048 00000020:00010000:2.0:1789760732.583292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.583295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.583298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.583302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.583306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.583316:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.583324:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.583352:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760732.583584:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23293952<>23298048 00000100:00100000:3.0:1789760732.584446:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493709824:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760732.619013:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760732.619031:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.619043:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.619064:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760732.619093:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493709824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760732.619243:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5687, 5687] 00000080:00200000:2.0:1789760732.619259:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.619264:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.619280:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23298048 i_blocks 47111 00000020:00200000:2.0:1789760732.619283:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.619292:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.619321:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23298048, count: 4096 00000080:00200000:2.0:1789760732.619329:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23298048, bytes: 4096 00000080:00200000:2.0:1789760732.619338:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.619346:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.619348:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23298048, 23302144) 0 0 00000080:00200000:2.0:1789760732.619353:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.619370:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5688, 5688] 00020000:00200000:2.0:1789760732.619378:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23298048 chunk: [23298048, 23302144) 23302144, 4096 00020000:00200000:2.0:1789760732.619383:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.620501:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5688, 5688] 00000008:00010000:3.0:1789760732.620533:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.620539:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760732.620553:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23298048, 23302144) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.620637:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23298048=0x1638000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.620705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.620708:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23302144 00000020:00010000:3.0:1789760732.620717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.620720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.620723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.620727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.620729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.620740:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.620748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.620834:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760732.620944:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 23298048<>23302144 00000100:00100000:3.0:1789760732.620993:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760732.621016:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493709952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.699066:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973007 gen 3 00000008:00010000:3.0:1789760732.699132:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760732.699141:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.699151:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.699170:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760732.699204:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493709952:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760732.768586:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5689, 5689] 00000080:00200000:1.0:1789760732.768605:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760732.768611:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760732.768628:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23306240 i_blocks 47111 00000020:00200000:1.0:1789760732.768632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760732.768644:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760732.768677:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23306240, count: 4096 00000080:00200000:1.0:1789760732.768684:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23306240, bytes: 4096 00000080:00200000:1.0:1789760732.768699:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760732.768707:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760732.768710:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23306240, 23310336) 0 0 00000080:00200000:1.0:1789760732.768714:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760732.768729:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5690, 5690] 00020000:00200000:1.0:1789760732.768736:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23306240 chunk: [23306240, 23310336) 23310336, 4096 00020000:00200000:1.0:1789760732.768740:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760732.768758:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5690, 5690] 00000008:00010000:1.0:1789760732.768776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760732.768782:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760732.768793:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23306240, 23310336) append 0 append lockpos 0 00000080:00200000:1.0:1789760732.768807:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23306240=0x163a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760732.768875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760732.768878:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23310336 00000020:00010000:1.0:1789760732.768885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760732.768889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760732.768893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760732.768896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760732.768899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760732.768909:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760732.768916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760732.768932:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760732.769033:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 23306240<>23310336 00000080:00200000:2.0:1789760732.814114:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5690, 5690] 00000080:00200000:2.0:1789760732.814130:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760732.814136:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760732.814151:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23310336 i_blocks 47111 00000020:00200000:2.0:1789760732.814155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760732.814164:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760732.814198:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23310336, count: 4096 00000080:00200000:2.0:1789760732.814205:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23310336, bytes: 4096 00000080:00200000:2.0:1789760732.814215:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760732.814568:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760732.814571:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23310336, 23314432) 0 0 00000080:00200000:2.0:1789760732.814576:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760732.814595:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5691, 5691] 00020000:00200000:2.0:1789760732.814603:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23310336 chunk: [23310336, 23314432) 23314432, 4096 00020000:00200000:2.0:1789760732.814608:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760732.814938:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5691, 5691] 00000008:00010000:2.0:1789760732.814959:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760732.814965:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760732.814977:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23310336, 23314432) append 0 append lockpos 0 00000080:00200000:2.0:1789760732.814990:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23310336=0x163b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760732.815116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760732.815119:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23314432 00000020:00010000:2.0:1789760732.815129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760732.815133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.815136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760732.815139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760732.815142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760732.815153:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760732.815161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.815184:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760732.815295:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23310336<>23314432 00000100:00100000:2.0:1789760732.815348:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760732.815373:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493710336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760732.879440:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973010 gen 3 00000008:00010000:3.0:1789760732.879493:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760732.879502:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.879514:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.879540:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760732.879581:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493710336:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760732.944371:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5692, 5692] 00000080:00200000:3.0:1789760732.944389:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.944432:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.944447:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23318528 i_blocks 47111 00000020:00200000:3.0:1789760732.944461:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.944473:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.944509:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23318528, count: 4096 00000080:00200000:3.0:1789760732.944516:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23318528, bytes: 4096 00000080:00200000:3.0:1789760732.944525:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.944534:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.944537:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23318528, 23322624) 0 0 00000080:00200000:3.0:1789760732.944541:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.944557:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5693, 5693] 00020000:00200000:3.0:1789760732.944565:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23318528 chunk: [23318528, 23322624) 23322624, 4096 00020000:00200000:3.0:1789760732.944569:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.944588:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5693, 5693] 00000008:00010000:3.0:1789760732.944605:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.944612:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760732.944623:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23318528, 23322624) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.944638:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23318528=0x163d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.944698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.944701:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23322624 00000020:00010000:3.0:1789760732.944710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.944714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.944717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.944720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.944723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.944730:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.944737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.944753:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760732.944997:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 23318528<>23322624 00000100:00100000:2.0:1789760732.945069:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493710592:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760732.991099:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760732.991114:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760732.991126:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760732.991145:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760732.991172:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493710592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760732.993195:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493710720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.048950:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973013 gen 3 00000008:00010000:1.0:1789760733.049029:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760733.049037:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.049057:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.049078:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760733.049115:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493710720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.050106:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493710848:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.084017:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973014 gen 3 00000008:00010000:1.0:1789760733.084065:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760733.084074:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.084084:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.084106:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760733.084136:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493710848:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.085083:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493710976:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.126512:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973015 gen 3 00000008:00010000:1.0:1789760733.126561:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760733.126569:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.126579:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.126601:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760733.126633:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493710976:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.127387:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711104:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.172158:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973016 gen 3 00000008:00010000:1.0:1789760733.172228:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760733.172236:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.172248:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.172270:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760733.172299:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711104:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.173439:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760733.173469:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711232:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.205852:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973017 gen 3 00000008:00010000:1.0:1789760733.206004:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760733.206012:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.206023:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.206102:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760733.206133:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711232:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.206981:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760733.207001:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711360:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.270114:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973018 gen 3 00000008:00010000:1.0:1789760733.270168:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760733.270177:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.270188:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.270289:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760733.270319:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711360:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.273778:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711488:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.314059:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973019 gen 3 00000008:00010000:1.0:1789760733.314113:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760733.314123:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.314135:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.314158:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760733.314188:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711488:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.316175:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711616:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.390909:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973020 gen 3 00000008:00010000:1.0:1789760733.390962:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760733.390971:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.390983:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.391009:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760733.391043:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493711616:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.391859:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711744:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760733.420655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760733.459685:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973021 gen 3 00000008:00010000:1.0:1789760733.459740:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760733.459749:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.459761:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.459783:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760733.459826:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493711744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.512321:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5703, 5703] 00000080:00200000:2.0:1789760733.512337:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.512343:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.512358:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23363584 i_blocks 47111 00000020:00200000:2.0:1789760733.512361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.512372:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.512445:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23363584, count: 4096 00000080:00200000:2.0:1789760733.512454:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23363584, bytes: 4096 00000080:00200000:2.0:1789760733.512463:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.512472:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.512474:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23363584, 23367680) 0 0 00000080:00200000:2.0:1789760733.512479:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.512497:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5704, 5704] 00020000:00200000:2.0:1789760733.512505:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23363584 chunk: [23363584, 23367680) 23367680, 4096 00020000:00200000:2.0:1789760733.512510:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.512527:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5704, 5704] 00000008:00010000:2.0:1789760733.512545:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.512551:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.512563:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23363584, 23367680) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.512577:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23363584=0x1648000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.512635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.512638:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23367680 00000020:00010000:2.0:1789760733.512646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.512650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.512654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.512657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.512659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.512668:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.512674:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.512689:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760733.512790:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23363584<>23367680 00000100:00100000:3.0:1789760733.512997:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712000:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760733.575981:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760733.575993:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.576002:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.576024:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760733.576048:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.576201:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5704, 5704] 00000080:00200000:2.0:1789760733.576213:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.576218:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.576229:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23367680 i_blocks 47111 00000020:00200000:2.0:1789760733.576231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.576240:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.576265:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23367680, count: 4096 00000080:00200000:2.0:1789760733.576270:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23367680, bytes: 4096 00000080:00200000:2.0:1789760733.576280:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.576287:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.576289:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23367680, 23371776) 0 0 00000080:00200000:2.0:1789760733.576293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.576305:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5705, 5705] 00020000:00200000:2.0:1789760733.576311:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23367680 chunk: [23367680, 23371776) 23371776, 4096 00020000:00200000:2.0:1789760733.576315:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.576329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5705, 5705] 00000008:00010000:2.0:1789760733.576342:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.576347:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.576356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23367680, 23371776) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.576366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23367680=0x1649000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.576527:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.576530:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23371776 00000020:00010000:2.0:1789760733.576537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.576540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.576544:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.576546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.576549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.576555:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.576560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.576576:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760733.576661:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23367680<>23371776 00000100:00100000:3.0:1789760733.576827:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.635774:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973024 gen 3 00000008:00010000:3.0:1789760733.635834:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760733.635841:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.635851:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.635975:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760733.636004:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712128:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.639104:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493712256:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760733.696619:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973025 gen 3 00000008:00010000:1.0:1789760733.696672:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760733.696681:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.696694:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.696716:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760733.696747:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493712256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.696847:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5706, 5706] 00000080:00200000:2.0:1789760733.696864:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.696870:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.696885:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23375872 i_blocks 47111 00000020:00200000:2.0:1789760733.696889:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.696972:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.697007:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23375872, count: 4096 00000080:00200000:2.0:1789760733.697023:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23375872, bytes: 4096 00000080:00200000:2.0:1789760733.697034:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.697042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.697044:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23375872, 23379968) 0 0 00000080:00200000:2.0:1789760733.697049:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.697064:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5707, 5707] 00020000:00200000:2.0:1789760733.697072:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23375872 chunk: [23375872, 23379968) 23379968, 4096 00020000:00200000:2.0:1789760733.697077:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.697097:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5707, 5707] 00000008:00010000:2.0:1789760733.697115:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.697122:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.697135:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23375872, 23379968) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.697148:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23375872=0x164b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.697210:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.697213:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23379968 00000020:00010000:2.0:1789760733.697222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.697226:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.697230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.697233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.697236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.697245:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.697252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.697266:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760733.697368:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23375872<>23379968 00000100:00100000:3.0:1789760733.697452:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.735009:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973026 gen 3 00000008:00010000:3.0:1789760733.735060:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760733.735068:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.735078:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.735097:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760733.735125:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.735283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5707, 5707] 00000080:00200000:2.0:1789760733.735300:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.735306:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.735321:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23379968 i_blocks 47111 00000020:00200000:2.0:1789760733.735325:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.735335:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.735382:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23379968, count: 4096 00000080:00200000:2.0:1789760733.735428:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23379968, bytes: 4096 00000080:00200000:2.0:1789760733.735441:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.735462:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.735464:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23379968, 23384064) 0 0 00000080:00200000:2.0:1789760733.735469:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.735485:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5708, 5708] 00020000:00200000:2.0:1789760733.735493:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23379968 chunk: [23379968, 23384064) 23384064, 4096 00020000:00200000:2.0:1789760733.735498:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.735517:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5708, 5708] 00000008:00010000:2.0:1789760733.735535:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.735541:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.735553:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23379968, 23384064) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.735567:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23379968=0x164c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.735625:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.735628:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23384064 00000020:00010000:2.0:1789760733.735636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.735640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.735644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.735647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.735650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.735657:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.735664:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.735680:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760733.735781:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23379968<>23384064 00000100:00100000:3.0:1789760733.736051:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.775423:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973027 gen 3 00000008:00010000:3.0:1789760733.775487:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760733.775495:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.775507:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.775524:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760733.775552:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.775697:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5708, 5708] 00000080:00200000:2.0:1789760733.775713:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.775719:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.775735:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23384064 i_blocks 47111 00000020:00200000:2.0:1789760733.775739:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.775749:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.775782:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23384064, count: 4096 00000080:00200000:2.0:1789760733.775789:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23384064, bytes: 4096 00000080:00200000:2.0:1789760733.775797:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.775806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.775808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23384064, 23388160) 0 0 00000080:00200000:2.0:1789760733.775813:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.775829:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5709, 5709] 00020000:00200000:2.0:1789760733.775837:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23384064 chunk: [23384064, 23388160) 23388160, 4096 00020000:00200000:2.0:1789760733.775842:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.775861:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5709, 5709] 00000008:00010000:2.0:1789760733.775880:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.775886:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.775909:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23384064, 23388160) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.775924:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23384064=0x164d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.775991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.775994:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23388160 00000020:00010000:2.0:1789760733.776002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.776006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.776010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.776013:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.776016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.776024:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.776031:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.776046:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760733.776146:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23384064<>23388160 00000100:00100000:3.0:1789760733.776530:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760733.776550:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712640:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760733.819921:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760733.819938:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.819949:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.819970:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760733.820000:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.820263:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5709, 5709] 00000080:00200000:2.0:1789760733.820278:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.820284:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.820301:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23388160 i_blocks 47111 00000020:00200000:2.0:1789760733.820305:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.820315:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.820361:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23388160, count: 4096 00000080:00200000:2.0:1789760733.820370:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23388160, bytes: 4096 00000080:00200000:2.0:1789760733.820380:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.820432:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.820434:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23388160, 23392256) 0 0 00000080:00200000:2.0:1789760733.820439:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.820454:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5710, 5710] 00020000:00200000:2.0:1789760733.820463:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23388160 chunk: [23388160, 23392256) 23392256, 4096 00020000:00200000:2.0:1789760733.820467:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.820486:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5710, 5710] 00000008:00010000:2.0:1789760733.820503:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.820509:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.820520:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23388160, 23392256) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.820535:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23388160=0x164e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.820597:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.820600:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23392256 00000020:00010000:2.0:1789760733.820610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.820613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.820617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.820621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.820624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.820632:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.820640:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.820655:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760733.820760:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23388160<>23392256 00000100:00100000:3.0:1789760733.821450:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.884895:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973029 gen 3 00000008:00010000:3.0:1789760733.884945:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760733.884951:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.884960:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.885015:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760733.885042:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493712768:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.885173:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5710, 5710] 00000080:00200000:2.0:1789760733.885191:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.885196:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.885208:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23392256 i_blocks 47111 00000020:00200000:2.0:1789760733.885212:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.885221:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.885249:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23392256, count: 4096 00000080:00200000:2.0:1789760733.885255:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23392256, bytes: 4096 00000080:00200000:2.0:1789760733.885263:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.885271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.885273:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23392256, 23396352) 0 0 00000080:00200000:2.0:1789760733.885277:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.885356:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5711, 5711] 00020000:00200000:2.0:1789760733.885362:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23392256 chunk: [23392256, 23396352) 23396352, 4096 00020000:00200000:2.0:1789760733.885366:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.885381:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5711, 5711] 00000008:00010000:2.0:1789760733.885425:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.885430:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.885954:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23392256, 23396352) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.885967:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23392256=0x164f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.886018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.886020:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23396352 00000020:00010000:2.0:1789760733.886028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.886031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.886035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.886037:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.886040:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.886046:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.886052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.886071:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760733.886215:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23392256<>23396352 00000100:00100000:3.0:1789760733.886349:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712896:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.955898:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973030 gen 3 00000008:00010000:3.0:1789760733.955939:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760733.955945:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.955955:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.955973:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760733.956001:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493712896:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.956143:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5711, 5711] 00000080:00200000:2.0:1789760733.956156:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.956161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.956171:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23396352 i_blocks 47111 00000020:00200000:2.0:1789760733.956175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.956182:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.956212:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23396352, count: 4096 00000080:00200000:2.0:1789760733.956217:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23396352, bytes: 4096 00000080:00200000:2.0:1789760733.956225:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.956232:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.956234:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23396352, 23400448) 0 0 00000080:00200000:2.0:1789760733.956238:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.956252:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5712, 5712] 00020000:00200000:2.0:1789760733.956259:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23396352 chunk: [23396352, 23400448) 23400448, 4096 00020000:00200000:2.0:1789760733.956263:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.956277:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5712, 5712] 00000008:00010000:2.0:1789760733.956535:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.956541:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.956550:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23396352, 23400448) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.956561:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23396352=0x1650000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.956602:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.956605:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23400448 00000020:00010000:2.0:1789760733.956611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.956614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.956617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.956620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.956623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.956629:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.956635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.956650:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760733.956727:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23396352<>23400448 00000100:00100000:3.0:1789760733.957217:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760733.989609:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973031 gen 3 00000008:00010000:3.0:1789760733.989644:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760733.989650:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760733.989659:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760733.989675:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760733.989700:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760733.989881:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5712, 5712] 00000080:00200000:2.0:1789760733.989901:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.989907:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.989920:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23400448 i_blocks 47111 00000020:00200000:2.0:1789760733.989923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.989932:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.989961:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23400448, count: 4096 00000080:00200000:2.0:1789760733.989967:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23400448, bytes: 4096 00000080:00200000:2.0:1789760733.989975:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.989982:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.989984:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23400448, 23404544) 0 0 00000080:00200000:2.0:1789760733.989988:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.990001:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5713, 5713] 00020000:00200000:2.0:1789760733.990009:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23400448 chunk: [23400448, 23404544) 23404544, 4096 00020000:00200000:2.0:1789760733.990012:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.990028:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5713, 5713] 00000008:00010000:2.0:1789760733.990040:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.990046:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760733.990054:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23400448, 23404544) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.990066:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23400448=0x1651000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.990109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.990122:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23404544 00000020:00010000:2.0:1789760733.990129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.990132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.990135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.990138:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.990140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.990147:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.990152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.990170:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760733.990248:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23400448<>23404544 00000100:00100000:3.0:1789760733.990499:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760733.990514:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.027274:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973032 gen 3 00000008:00010000:3.0:1789760734.027308:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760734.027314:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.027324:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.027342:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760734.027378:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.027548:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5713, 5713] 00000080:00200000:2.0:1789760734.027560:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.027565:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.027577:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23404544 i_blocks 47111 00000020:00200000:2.0:1789760734.027580:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.027587:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.027613:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23404544, count: 4096 00000080:00200000:2.0:1789760734.027619:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23404544, bytes: 4096 00000080:00200000:2.0:1789760734.027627:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.027633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.027635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23404544, 23408640) 0 0 00000080:00200000:2.0:1789760734.027639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.027651:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5714, 5714] 00020000:00200000:2.0:1789760734.027657:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23404544 chunk: [23404544, 23408640) 23408640, 4096 00020000:00200000:2.0:1789760734.027660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.027674:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5714, 5714] 00000008:00010000:2.0:1789760734.027699:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.027704:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.027713:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23404544, 23408640) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.027725:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23404544=0x1652000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.027767:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.027770:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23408640 00000020:00010000:2.0:1789760734.027776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.027779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.027783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.027786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.027788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.027794:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.027799:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.027812:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760734.027882:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23404544<>23408640 00000100:00100000:3.0:1789760734.029223:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713280:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760734.063269:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760734.063280:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.063291:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.063309:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760734.063334:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.063986:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5714, 5714] 00000080:00200000:2.0:1789760734.063999:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.064004:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.064015:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23408640 i_blocks 47111 00000020:00200000:2.0:1789760734.064018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.064027:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.064052:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23408640, count: 4096 00000080:00200000:2.0:1789760734.064057:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23408640, bytes: 4096 00000080:00200000:2.0:1789760734.064066:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.064073:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.064075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23408640, 23412736) 0 0 00000080:00200000:2.0:1789760734.064079:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.064091:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5715, 5715] 00020000:00200000:2.0:1789760734.064097:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23408640 chunk: [23408640, 23412736) 23412736, 4096 00020000:00200000:2.0:1789760734.064102:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.064116:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5715, 5715] 00000008:00010000:2.0:1789760734.064127:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.064132:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.064141:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23408640, 23412736) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.064151:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23408640=0x1653000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.064191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.064194:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23412736 00000020:00010000:2.0:1789760734.064200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.064203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.064207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.064210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.064213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.064219:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.064224:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.064236:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760734.064311:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23408640<>23412736 00000100:00100000:2.0:1789760734.064382:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493713408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.105069:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973033 gen 3 00000100:00100000:3.0:1789760734.105100:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f82a00 x1876699493713408/t4294973034(4294973034) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760750 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760734.105138:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760734.105144:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.105153:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.105245:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760734.105267:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493713408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.108204:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5715, 5715] 00000080:00200000:2.0:1789760734.108224:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.108229:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.108242:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23412736 i_blocks 47111 00000020:00200000:2.0:1789760734.108246:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.108258:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.108287:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23412736, count: 4096 00000080:00200000:2.0:1789760734.108293:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23412736, bytes: 4096 00000080:00200000:2.0:1789760734.108304:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.108311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.108313:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23412736, 23416832) 0 0 00000080:00200000:2.0:1789760734.108316:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.108328:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5716, 5716] 00020000:00200000:2.0:1789760734.108335:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23412736 chunk: [23412736, 23416832) 23416832, 4096 00020000:00200000:2.0:1789760734.108339:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.108356:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5716, 5716] 00000008:00010000:2.0:1789760734.108381:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.108410:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.108420:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23412736, 23416832) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.108431:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23412736=0x1654000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.108485:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.108488:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23416832 00000020:00010000:2.0:1789760734.108495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.108498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.108501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.108504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.108507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.108513:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.108518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.108543:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760734.108623:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23412736<>23416832 00000100:00100000:3.0:1789760734.108863:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713536:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.143014:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973035 gen 3 00000008:00010000:3.0:1789760734.143062:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760734.143069:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.143076:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.143096:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760734.143130:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.143266:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5716, 5716] 00000080:00200000:2.0:1789760734.143278:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.143283:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.143296:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23416832 i_blocks 47111 00000020:00200000:2.0:1789760734.143299:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.143308:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.143333:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23416832, count: 4096 00000080:00200000:2.0:1789760734.143339:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23416832, bytes: 4096 00000080:00200000:2.0:1789760734.143348:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.143355:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.143357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23416832, 23420928) 0 0 00000080:00200000:2.0:1789760734.143362:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.143415:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5717, 5717] 00020000:00200000:2.0:1789760734.143423:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23416832 chunk: [23416832, 23420928) 23420928, 4096 00020000:00200000:2.0:1789760734.143427:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.143442:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5717, 5717] 00000008:00010000:2.0:1789760734.143456:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.143462:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.143473:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23416832, 23420928) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.143484:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23416832=0x1655000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.143527:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.143530:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23420928 00000020:00010000:2.0:1789760734.143537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.143541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.143544:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.143547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.143550:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.143556:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.143561:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.143574:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760734.143654:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23416832<>23420928 00000100:00100000:3.0:1789760734.143844:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760734.143858:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.228447:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973036 gen 3 00000008:00010000:3.0:1789760734.228483:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760734.228489:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.228499:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.228518:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760734.228544:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713664:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.228643:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5717, 5717] 00000080:00200000:2.0:1789760734.228655:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.228660:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.228672:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23420928 i_blocks 47111 00000020:00200000:2.0:1789760734.228675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.228684:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.228721:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23420928, count: 4096 00000080:00200000:2.0:1789760734.228727:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23420928, bytes: 4096 00000080:00200000:2.0:1789760734.228736:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.228742:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.228745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23420928, 23425024) 0 0 00000080:00200000:2.0:1789760734.228748:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.228762:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5718, 5718] 00020000:00200000:2.0:1789760734.228768:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23420928 chunk: [23420928, 23425024) 23425024, 4096 00020000:00200000:2.0:1789760734.228772:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.228788:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5718, 5718] 00000008:00010000:2.0:1789760734.228801:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.228806:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.228814:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23420928, 23425024) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.228825:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23420928=0x1656000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.228868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.228870:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23425024 00000020:00010000:2.0:1789760734.228877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.228880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.228883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.228897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.228900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.228907:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.228912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.228926:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760734.229008:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23420928<>23425024 00000100:00100000:3.0:1789760734.229067:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713792:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.303727:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973037 gen 3 00000008:00010000:3.0:1789760734.303764:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760734.303770:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.303779:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.303798:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760734.303823:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493713792:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.303978:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5718, 5718] 00000080:00200000:2.0:1789760734.303992:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.303997:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.304010:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23425024 i_blocks 47111 00000020:00200000:2.0:1789760734.304014:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.304023:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.304051:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23425024, count: 4096 00000080:00200000:2.0:1789760734.304057:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23425024, bytes: 4096 00000080:00200000:2.0:1789760734.304064:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.304071:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.304074:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23425024, 23429120) 0 0 00000080:00200000:2.0:1789760734.304078:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.304089:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5719, 5719] 00020000:00200000:2.0:1789760734.304095:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23425024 chunk: [23425024, 23429120) 23429120, 4096 00020000:00200000:2.0:1789760734.304099:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.304124:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5719, 5719] 00000008:00010000:2.0:1789760734.304137:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.304142:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.304150:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23425024, 23429120) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.304160:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23425024=0x1657000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.304203:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.304205:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23429120 00000020:00010000:2.0:1789760734.304212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.304216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.304218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.304221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.304224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.304229:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.304236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.304250:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760734.304324:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23425024<>23429120 00000100:00100000:3.0:1789760734.304594:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493713920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.350190:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973038 gen 3 00000008:00010000:3.0:1789760734.350227:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760734.350234:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.350244:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.350261:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760734.350286:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493713920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.350492:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5719, 5719] 00000080:00200000:2.0:1789760734.350505:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.350510:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.350521:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23429120 i_blocks 47111 00000020:00200000:2.0:1789760734.350524:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.350532:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.350559:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23429120, count: 4096 00000080:00200000:2.0:1789760734.350564:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23429120, bytes: 4096 00000080:00200000:2.0:1789760734.350580:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.350589:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.350591:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23429120, 23433216) 0 0 00000080:00200000:2.0:1789760734.350595:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.350608:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5720, 5720] 00020000:00200000:2.0:1789760734.350615:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23429120 chunk: [23429120, 23433216) 23433216, 4096 00020000:00200000:2.0:1789760734.350619:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.350632:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5720, 5720] 00000008:00010000:2.0:1789760734.350646:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.350650:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.350659:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23429120, 23433216) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.350669:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23429120=0x1658000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.350723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.350726:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23433216 00000020:00010000:2.0:1789760734.350732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.350736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.350739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.350742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.350745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.350751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.350756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.350769:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760734.350841:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23429120<>23433216 00000100:00100000:3.0:1789760734.351179:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760734.351192:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714048:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760734.460436:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760734.460447:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.460456:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.460474:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760734.460497:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714048:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.460585:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5720, 5720] 00000080:00200000:2.0:1789760734.460597:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.460602:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.460614:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23433216 i_blocks 47111 00000020:00200000:2.0:1789760734.460617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.460625:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.460652:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23433216, count: 4096 00000080:00200000:2.0:1789760734.460657:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23433216, bytes: 4096 00000080:00200000:2.0:1789760734.460664:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.460670:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.460672:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23433216, 23437312) 0 0 00000080:00200000:2.0:1789760734.460676:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.460688:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5721, 5721] 00020000:00200000:2.0:1789760734.460695:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23433216 chunk: [23433216, 23437312) 23437312, 4096 00020000:00200000:2.0:1789760734.460710:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.460725:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5721, 5721] 00000008:00010000:2.0:1789760734.460738:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.460742:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.460750:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23433216, 23437312) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.460761:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23433216=0x1659000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.460802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.460804:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23437312 00000020:00010000:2.0:1789760734.460811:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.460815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.460817:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.460820:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.460822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.460828:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.460833:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.460847:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760734.461189:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23433216<>23437312 00000100:00100000:3.0:1789760734.461346:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.545779:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973040 gen 3 00000008:00010000:3.0:1789760734.545822:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760734.545828:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.545836:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.545858:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760734.545899:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.546276:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5721, 5721] 00000080:00200000:2.0:1789760734.546288:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.546293:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.546308:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23437312 i_blocks 47111 00000020:00200000:2.0:1789760734.546312:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.546453:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.546479:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23437312, count: 4096 00000080:00200000:2.0:1789760734.546485:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23437312, bytes: 4096 00000080:00200000:2.0:1789760734.546495:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.546502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.546505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23437312, 23441408) 0 0 00000080:00200000:2.0:1789760734.546509:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.546522:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5722, 5722] 00020000:00200000:2.0:1789760734.546528:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23437312 chunk: [23437312, 23441408) 23441408, 4096 00020000:00200000:2.0:1789760734.546532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.546548:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5722, 5722] 00000008:00010000:2.0:1789760734.546562:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.546566:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.546575:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23437312, 23441408) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.546586:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23437312=0x165a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.546629:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.546632:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23441408 00000020:00010000:2.0:1789760734.546639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.546643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.546645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.546649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.546651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.546658:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.546663:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.546690:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760734.546766:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23437312<>23441408 00000100:00100000:3.0:1789760734.547095:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760734.547124:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714304:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760734.592277:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760734.592293:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.592304:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.592426:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760734.592454:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714304:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.592549:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5722, 5722] 00000080:00200000:2.0:1789760734.592563:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.592569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.592580:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23441408 i_blocks 47111 00000020:00200000:2.0:1789760734.592583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.592593:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.592620:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23441408, count: 4096 00000080:00200000:2.0:1789760734.592813:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23441408, bytes: 4096 00000080:00200000:2.0:1789760734.592825:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.592833:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.592835:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23441408, 23445504) 0 0 00000080:00200000:2.0:1789760734.592839:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.592852:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5723, 5723] 00020000:00200000:2.0:1789760734.592859:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23441408 chunk: [23441408, 23445504) 23445504, 4096 00020000:00200000:2.0:1789760734.592982:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.592999:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5723, 5723] 00000008:00010000:2.0:1789760734.593014:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.593318:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.593330:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23441408, 23445504) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.593342:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23441408=0x165b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.593498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.593501:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23445504 00000020:00010000:2.0:1789760734.593509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.593512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.593515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.593518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.593520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.593527:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.593533:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.593552:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760734.593842:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23441408<>23445504 00000100:00100000:3.0:1789760734.593982:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.646452:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973041 gen 3 00000100:00100000:3.0:1789760734.646491:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81880 x1876699493714432/t4294973042(4294973042) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760750 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760734.646530:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760734.646535:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.646545:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.646563:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760734.646589:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.646978:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5723, 5723] 00000080:00200000:2.0:1789760734.646991:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.646996:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.647007:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23445504 i_blocks 47111 00000020:00200000:2.0:1789760734.647010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.647019:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.647045:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23445504, count: 4096 00000080:00200000:2.0:1789760734.647052:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23445504, bytes: 4096 00000080:00200000:2.0:1789760734.647061:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.647068:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.647069:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23445504, 23449600) 0 0 00000080:00200000:2.0:1789760734.647073:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.647085:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5724, 5724] 00020000:00200000:2.0:1789760734.647093:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23445504 chunk: [23445504, 23449600) 23449600, 4096 00020000:00200000:2.0:1789760734.647096:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.647587:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5724, 5724] 00000008:00010000:2.0:1789760734.647606:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.647610:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.647621:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23445504, 23449600) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.647632:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23445504=0x165c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.649006:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.649013:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23449600 00000020:00010000:2.0:1789760734.649023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.649027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.649030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.649033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.649035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.649044:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.649052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.649079:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760734.649278:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23445504<>23449600 00000100:00100000:3.0:1789760734.649429:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714560:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.749717:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973043 gen 3 00000008:00010000:3.0:1789760734.749776:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760734.749783:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.749792:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.749809:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760734.749834:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714560:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.749988:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5724, 5724] 00000080:00200000:2.0:1789760734.750000:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.750005:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.750016:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23449600 i_blocks 47111 00000020:00200000:2.0:1789760734.750020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.750028:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.750054:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23449600, count: 4096 00000080:00200000:2.0:1789760734.750060:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23449600, bytes: 4096 00000080:00200000:2.0:1789760734.750068:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.750075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.750077:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23449600, 23453696) 0 0 00000080:00200000:2.0:1789760734.750081:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.750094:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5725, 5725] 00020000:00200000:2.0:1789760734.750101:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23449600 chunk: [23449600, 23453696) 23453696, 4096 00020000:00200000:2.0:1789760734.750105:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.750121:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5725, 5725] 00000008:00010000:2.0:1789760734.750134:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.750138:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.750148:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23449600, 23453696) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.750157:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23449600=0x165d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.750198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.750201:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23453696 00000020:00010000:2.0:1789760734.750208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.750212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.750215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.750218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.750221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.750228:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.750233:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.750260:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760734.750337:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23449600<>23453696 00000100:00100000:3.0:1789760734.750752:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760734.750769:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714688:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760734.860018:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973044 gen 3 00000008:00010000:3.0:1789760734.860053:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760734.860059:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760734.860069:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760734.860091:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760734.860118:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714688:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760734.860223:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5725, 5725] 00000080:00200000:2.0:1789760734.860235:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.860240:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.860252:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23453696 i_blocks 47111 00000020:00200000:2.0:1789760734.860255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.860264:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.860290:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23453696, count: 4096 00000080:00200000:2.0:1789760734.860296:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23453696, bytes: 4096 00000080:00200000:2.0:1789760734.860305:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.860312:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.860314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23453696, 23457792) 0 0 00000080:00200000:2.0:1789760734.860318:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.860331:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5726, 5726] 00020000:00200000:2.0:1789760734.860338:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23453696 chunk: [23453696, 23457792) 23457792, 4096 00020000:00200000:2.0:1789760734.860342:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.860357:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5726, 5726] 00000008:00010000:2.0:1789760734.860370:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.860375:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760734.860440:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23453696, 23457792) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.860453:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23453696=0x165e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.860496:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.860499:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23457792 00000020:00010000:2.0:1789760734.860506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.860509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.860512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.860515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.860518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.860526:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.860533:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.860548:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760734.860628:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23453696<>23457792 00000100:00100000:3.0:1789760734.861005:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714816:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.019979:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973045 gen 3 00000008:00010000:3.0:1789760735.020015:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760735.020022:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.020031:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.020046:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760735.020068:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493714816:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.020160:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5726, 5726] 00000080:00200000:2.0:1789760735.020172:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.020177:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.020189:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23457792 i_blocks 47111 00000020:00200000:2.0:1789760735.020192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.020201:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.020226:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23457792, count: 4096 00000080:00200000:2.0:1789760735.020232:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23457792, bytes: 4096 00000080:00200000:2.0:1789760735.020239:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.020246:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.020247:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23457792, 23461888) 0 0 00000080:00200000:2.0:1789760735.020251:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.020263:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5727, 5727] 00020000:00200000:2.0:1789760735.020269:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23457792 chunk: [23457792, 23461888) 23461888, 4096 00020000:00200000:2.0:1789760735.020273:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.020285:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5727, 5727] 00000008:00010000:2.0:1789760735.020297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.020302:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.020309:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23457792, 23461888) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.020321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23457792=0x165f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.020358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.020360:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23461888 00000020:00010000:2.0:1789760735.020366:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.020370:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.020372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.020375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.021132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.021147:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.021155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.021175:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760735.021255:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23457792<>23461888 00000100:00100000:3.0:1789760735.021329:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.082452:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973046 gen 3 00000008:00010000:3.0:1789760735.082487:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760735.082494:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.082503:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.082533:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760735.082558:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493714944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.082657:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5727, 5727] 00000080:00200000:2.0:1789760735.082669:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.082675:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.082686:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23461888 i_blocks 47111 00000020:00200000:2.0:1789760735.082690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.082699:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.082725:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23461888, count: 4096 00000080:00200000:2.0:1789760735.082732:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23461888, bytes: 4096 00000080:00200000:2.0:1789760735.082739:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.082759:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.082761:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23461888, 23465984) 0 0 00000080:00200000:2.0:1789760735.082765:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.082777:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5728, 5728] 00020000:00200000:2.0:1789760735.082784:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23461888 chunk: [23461888, 23465984) 23465984, 4096 00020000:00200000:2.0:1789760735.082788:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.082802:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5728, 5728] 00000008:00010000:2.0:1789760735.082814:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.082819:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.082826:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23461888, 23465984) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.082836:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23461888=0x1660000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.082880:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.082883:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23465984 00000020:00010000:2.0:1789760735.082890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.082894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.082897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.082900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.082903:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.082909:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.082915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.082928:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760735.083000:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23461888<>23465984 00000100:00100000:2.0:1789760735.083045:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760735.083057:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.143494:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973047 gen 3 00000008:00010000:3.0:1789760735.143530:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760735.143537:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.143547:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.143568:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760735.143594:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.144034:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5728, 5728] 00000080:00200000:2.0:1789760735.144048:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.144053:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.144065:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23465984 i_blocks 47111 00000020:00200000:2.0:1789760735.144069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.144079:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.144105:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23465984, count: 4096 00000080:00200000:2.0:1789760735.144111:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23465984, bytes: 4096 00000080:00200000:2.0:1789760735.144120:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.144126:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.144128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23465984, 23470080) 0 0 00000080:00200000:2.0:1789760735.144132:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.144145:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5729, 5729] 00020000:00200000:2.0:1789760735.144152:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23465984 chunk: [23465984, 23470080) 23470080, 4096 00020000:00200000:2.0:1789760735.144155:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.144170:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5729, 5729] 00000008:00010000:2.0:1789760735.144183:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.144187:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.144196:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23465984, 23470080) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.144208:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23465984=0x1661000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.144559:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.144563:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23470080 00000020:00010000:2.0:1789760735.144571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.144574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.144578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.144582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.144585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.144592:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.144598:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.144614:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760735.144690:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23465984<>23470080 00000100:00100000:2.0:1789760735.144733:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760735.146557:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.188252:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973048 gen 3 00000008:00010000:3.0:1789760735.188289:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760735.188296:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.188305:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.188324:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760735.188349:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.188484:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5729, 5729] 00000080:00200000:2.0:1789760735.188498:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.188503:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.188515:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23470080 i_blocks 47111 00000020:00200000:2.0:1789760735.188518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.188528:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.188555:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23470080, count: 4096 00000080:00200000:2.0:1789760735.188561:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23470080, bytes: 4096 00000080:00200000:2.0:1789760735.188568:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.188575:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.188576:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23470080, 23474176) 0 0 00000080:00200000:2.0:1789760735.188580:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.188592:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5730, 5730] 00020000:00200000:2.0:1789760735.188599:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23470080 chunk: [23470080, 23474176) 23474176, 4096 00020000:00200000:2.0:1789760735.188603:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.188617:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5730, 5730] 00000008:00010000:2.0:1789760735.188631:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.188637:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.188646:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23470080, 23474176) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.188656:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23470080=0x1662000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.188704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.188707:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23474176 00000020:00010000:2.0:1789760735.188714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.188718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.188721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.188724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.188727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.188733:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.188739:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.188753:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760735.188830:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23470080<>23474176 00000100:00100000:3.0:1789760735.188986:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715328:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.225629:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973049 gen 3 00000008:00010000:3.0:1789760735.225664:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760735.225670:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.225678:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.225700:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760735.225724:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.225818:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5730, 5730] 00000080:00200000:2.0:1789760735.225831:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.225836:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.225847:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23474176 i_blocks 47111 00000020:00200000:2.0:1789760735.225851:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.225871:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.225897:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23474176, count: 4096 00000080:00200000:2.0:1789760735.225903:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23474176, bytes: 4096 00000080:00200000:2.0:1789760735.225911:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.225918:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.225920:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23474176, 23478272) 0 0 00000080:00200000:2.0:1789760735.225924:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.225937:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5731, 5731] 00020000:00200000:2.0:1789760735.225944:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23474176 chunk: [23474176, 23478272) 23478272, 4096 00020000:00200000:2.0:1789760735.225948:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.225962:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5731, 5731] 00000008:00010000:2.0:1789760735.225974:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.225979:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.225988:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23474176, 23478272) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.225999:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23474176=0x1663000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.226040:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.226043:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23478272 00000020:00010000:2.0:1789760735.226050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.226054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.226057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.226060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.226063:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.226068:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.226073:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.226088:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760735.226166:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23474176<>23478272 00000100:00100000:3.0:1789760735.226309:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715456:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760735.278854:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760735.278862:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.278872:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.278929:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760735.278953:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.279110:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5731, 5731] 00000080:00200000:2.0:1789760735.279122:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.279126:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.279138:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23478272 i_blocks 47111 00000020:00200000:2.0:1789760735.279141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.279149:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.279176:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23478272, count: 4096 00000080:00200000:2.0:1789760735.279182:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23478272, bytes: 4096 00000080:00200000:2.0:1789760735.279189:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.279195:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.279197:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23478272, 23482368) 0 0 00000080:00200000:2.0:1789760735.279201:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.279213:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5732, 5732] 00020000:00200000:2.0:1789760735.279220:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23478272 chunk: [23478272, 23482368) 23482368, 4096 00020000:00200000:2.0:1789760735.279223:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.279237:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5732, 5732] 00000008:00010000:2.0:1789760735.279250:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.279255:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.279264:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23478272, 23482368) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.279274:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23478272=0x1664000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.279316:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.279319:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23482368 00000020:00010000:2.0:1789760735.279325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.279328:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.279331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.279334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.279336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.279341:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.279347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.279363:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760735.279580:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23478272<>23482368 00000100:00100000:3.0:1789760735.280014:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715584:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.318176:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973051 gen 3 00000008:00010000:3.0:1789760735.318222:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760735.318229:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.318237:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.318257:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760735.318283:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.318590:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5732, 5732] 00000080:00200000:2.0:1789760735.318604:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.318608:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.318619:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23482368 i_blocks 47111 00000020:00200000:2.0:1789760735.318623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.318633:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.318660:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23482368, count: 4096 00000080:00200000:2.0:1789760735.318666:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23482368, bytes: 4096 00000080:00200000:2.0:1789760735.318674:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.318682:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.318684:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23482368, 23486464) 0 0 00000080:00200000:2.0:1789760735.318688:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.318701:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5733, 5733] 00020000:00200000:2.0:1789760735.318707:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23482368 chunk: [23482368, 23486464) 23486464, 4096 00020000:00200000:2.0:1789760735.318711:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.318726:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5733, 5733] 00000008:00010000:2.0:1789760735.318739:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.318745:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.318754:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23482368, 23486464) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.318764:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23482368=0x1665000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.318809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.318812:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23486464 00000020:00010000:2.0:1789760735.318833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.318837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.318840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.318843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.318846:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.318852:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.318968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.318984:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760735.319057:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23482368<>23486464 00000100:00100000:3.0:1789760735.319158:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.371852:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973052 gen 3 00000008:00010000:3.0:1789760735.371899:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760735.371906:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.371916:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.372030:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5733, 5733] 00000080:00200000:2.0:1789760735.372045:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.372050:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.372062:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23486464 i_blocks 47111 00000020:00200000:2.0:1789760735.372065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.372075:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.372105:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23486464, count: 4096 00000080:00200000:2.0:1789760735.372111:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23486464, bytes: 4096 00000080:00200000:2.0:1789760735.372119:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.372127:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.372130:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23486464, 23490560) 0 0 00000080:00200000:2.0:1789760735.372134:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.372147:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5734, 5734] 00020000:00200000:2.0:1789760735.372155:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23486464 chunk: [23486464, 23490560) 23490560, 4096 00020000:00200000:2.0:1789760735.372159:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.372177:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5734, 5734] 00000008:00010000:2.0:1789760735.372192:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.372197:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.372207:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23486464, 23490560) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.372219:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23486464=0x1666000, pages 1 (max 131072), lockless, parallel 00000008:00010000:3.0:1789760735.372269:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760735.372299:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715712:192.168.203.160@tcp:4:dd.0 00000020:00200000:2.0:1789760735.373028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.373033:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23490560 00000020:00010000:2.0:1789760735.373041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.373045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.373048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.373052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.373055:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.373063:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.373069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.373088:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760735.373168:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23486464<>23490560 00000100:00100000:3.0:1789760735.373235:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715840:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760735.420516:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760735.420527:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.420536:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.420552:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760735.420575:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493715840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.420648:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5734, 5734] 00000080:00200000:2.0:1789760735.420660:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.420665:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.420676:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23490560 i_blocks 47111 00000020:00200000:2.0:1789760735.420680:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.420689:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.420717:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23490560, count: 4096 00000080:00200000:2.0:1789760735.420723:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23490560, bytes: 4096 00000080:00200000:2.0:1789760735.420731:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.420738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.420740:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23490560, 23494656) 0 0 00000080:00200000:2.0:1789760735.420744:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.420758:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5735, 5735] 00020000:00200000:2.0:1789760735.420765:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23490560 chunk: [23490560, 23494656) 23494656, 4096 00020000:00200000:2.0:1789760735.420769:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.420783:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5735, 5735] 00000008:00010000:2.0:1789760735.420795:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.420801:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.420809:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23490560, 23494656) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.420829:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23490560=0x1667000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.420874:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.420877:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23494656 00000020:00010000:2.0:1789760735.420883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.420886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.420889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.420892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.420895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.420900:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.420905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.420917:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760735.421146:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23490560<>23494656 00000100:00100000:3.0:1789760735.421213:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715968:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.470192:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973054 gen 3 00000008:00010000:3.0:1789760735.470235:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760735.470241:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.470249:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.470266:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760735.470291:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493715968:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.470553:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5735, 5735] 00000080:00200000:2.0:1789760735.470565:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.470570:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.470581:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23494656 i_blocks 47111 00000020:00200000:2.0:1789760735.470585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.470594:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.470619:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23494656, count: 4096 00000080:00200000:2.0:1789760735.470625:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23494656, bytes: 4096 00000080:00200000:2.0:1789760735.470634:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.470641:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.470644:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23494656, 23498752) 0 0 00000080:00200000:2.0:1789760735.470648:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.470661:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5736, 5736] 00020000:00200000:2.0:1789760735.470667:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23494656 chunk: [23494656, 23498752) 23498752, 4096 00020000:00200000:2.0:1789760735.470670:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.470685:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5736, 5736] 00000008:00010000:2.0:1789760735.470698:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.470704:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.470712:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23494656, 23498752) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.470723:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23494656=0x1668000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.470766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.470768:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23498752 00000020:00010000:2.0:1789760735.470776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.470779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.470782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.470785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.470788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.470795:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.470801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.470814:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760735.470906:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23494656<>23498752 00000100:00100000:3.0:1789760735.470970:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.509631:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973055 gen 3 00000008:00010000:3.0:1789760735.509666:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760735.509672:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.509682:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.509699:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760735.509723:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716096:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.510008:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5736, 5736] 00000080:00200000:2.0:1789760735.510021:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.510026:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.510038:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23498752 i_blocks 47111 00000020:00200000:2.0:1789760735.510042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.510051:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.510077:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23498752, count: 4096 00000080:00200000:2.0:1789760735.510083:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23498752, bytes: 4096 00000080:00200000:2.0:1789760735.510091:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.510098:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.510100:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23498752, 23502848) 0 0 00000080:00200000:2.0:1789760735.510104:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.510116:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5737, 5737] 00020000:00200000:2.0:1789760735.510122:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23498752 chunk: [23498752, 23502848) 23502848, 4096 00020000:00200000:2.0:1789760735.510126:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.510140:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5737, 5737] 00000008:00010000:2.0:1789760735.510154:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.510160:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.510169:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23498752, 23502848) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.510179:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23498752=0x1669000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.510218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.510221:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23502848 00000020:00010000:2.0:1789760735.510228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.510231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.510234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.510238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.510240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.510246:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.510252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.510266:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760735.510340:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23498752<>23502848 00000100:00100000:3.0:1789760735.510985:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716224:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760735.547258:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760735.547269:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.547278:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.547295:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760735.547322:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716224:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.547417:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5737, 5737] 00000080:00200000:2.0:1789760735.547436:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.547442:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.547453:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23502848 i_blocks 47111 00000020:00200000:2.0:1789760735.547457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.547466:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.547494:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23502848, count: 4096 00000080:00200000:2.0:1789760735.547501:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23502848, bytes: 4096 00000080:00200000:2.0:1789760735.547509:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.547516:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.547519:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23502848, 23506944) 0 0 00000080:00200000:2.0:1789760735.547522:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.547535:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5738, 5738] 00020000:00200000:2.0:1789760735.547542:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23502848 chunk: [23502848, 23506944) 23506944, 4096 00020000:00200000:2.0:1789760735.547546:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.547561:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5738, 5738] 00000008:00010000:2.0:1789760735.547574:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.547580:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.547589:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23502848, 23506944) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.547599:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23502848=0x166a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.547642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.547644:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23506944 00000020:00010000:2.0:1789760735.547651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.547654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.547657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.547660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.547663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.547670:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.547675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.547690:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760735.547776:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23502848<>23506944 00000100:00100000:3.0:1789760735.547850:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716352:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.589370:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973057 gen 3 00000008:00010000:3.0:1789760735.589447:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760735.589453:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.589462:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.589529:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760735.589553:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716352:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.589631:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5738, 5738] 00000080:00200000:2.0:1789760735.589643:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.589648:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.589660:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23506944 i_blocks 47111 00000020:00200000:2.0:1789760735.589664:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.589673:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.589700:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23506944, count: 4096 00000080:00200000:2.0:1789760735.589706:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23506944, bytes: 4096 00000080:00200000:2.0:1789760735.589714:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.589721:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.589723:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23506944, 23511040) 0 0 00000080:00200000:2.0:1789760735.589727:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.589741:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5739, 5739] 00020000:00200000:2.0:1789760735.589747:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23506944 chunk: [23506944, 23511040) 23511040, 4096 00020000:00200000:2.0:1789760735.589751:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.589764:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5739, 5739] 00000008:00010000:2.0:1789760735.589777:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.589782:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.589791:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23506944, 23511040) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.589801:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23506944=0x166b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.589843:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.589846:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23511040 00000020:00010000:2.0:1789760735.589852:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.589856:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.589858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.589861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.589864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.589870:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.589876:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.589890:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760735.590556:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23506944<>23511040 00000100:00100000:3.0:1789760735.590637:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.677255:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973058 gen 3 00000008:00010000:3.0:1789760735.677294:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760735.677301:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.677311:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.677329:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760735.677354:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716480:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.677486:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5739, 5739] 00000080:00200000:2.0:1789760735.677499:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.677504:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.677516:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23511040 i_blocks 47111 00000020:00200000:2.0:1789760735.677520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.677529:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.677554:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23511040, count: 4096 00000080:00200000:2.0:1789760735.677561:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23511040, bytes: 4096 00000080:00200000:2.0:1789760735.677569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.677576:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.677579:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23511040, 23515136) 0 0 00000080:00200000:2.0:1789760735.677582:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.677594:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5740, 5740] 00020000:00200000:2.0:1789760735.677601:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23511040 chunk: [23511040, 23515136) 23515136, 4096 00020000:00200000:2.0:1789760735.677604:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.677618:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5740, 5740] 00000008:00010000:2.0:1789760735.677631:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.677636:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.677645:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23511040, 23515136) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.677656:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23511040=0x166c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.677698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.677700:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23515136 00000020:00010000:2.0:1789760735.677707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.677710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.677713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.677716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.677719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.677726:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.677732:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.677745:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760735.677820:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23511040<>23515136 00000100:00100000:3.0:1789760735.678213:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760735.678228:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716608:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.753794:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973059 gen 3 00000008:00010000:3.0:1789760735.753829:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760735.753836:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.753846:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.753866:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760735.753891:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716608:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.754055:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5740, 5740] 00000080:00200000:2.0:1789760735.754067:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.754072:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.754084:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23515136 i_blocks 47111 00000020:00200000:2.0:1789760735.754088:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.754096:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.754121:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23515136, count: 4096 00000080:00200000:2.0:1789760735.754127:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23515136, bytes: 4096 00000080:00200000:2.0:1789760735.754136:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.754143:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.754145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23515136, 23519232) 0 0 00000080:00200000:2.0:1789760735.754149:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.754161:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5741, 5741] 00020000:00200000:2.0:1789760735.754168:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23515136 chunk: [23515136, 23519232) 23519232, 4096 00020000:00200000:2.0:1789760735.754172:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.754186:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5741, 5741] 00000008:00010000:2.0:1789760735.754198:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.754203:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.754212:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23515136, 23519232) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.754223:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23515136=0x166d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.754265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.754268:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23519232 00000020:00010000:2.0:1789760735.754275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.754278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.754281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.754284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.754286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.754292:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.754298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.754324:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760735.754463:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23515136<>23519232 00000100:00100000:3.0:1789760735.754605:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.804289:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973060 gen 3 00000008:00010000:3.0:1789760735.804320:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760735.804350:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.804360:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.804383:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760735.804530:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493716736:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760735.804621:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5741, 5741] 00000080:00200000:2.0:1789760735.804634:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.804639:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.804650:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23519232 i_blocks 47111 00000020:00200000:2.0:1789760735.804654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.804663:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.804690:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23519232, count: 4096 00000080:00200000:2.0:1789760735.804695:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23519232, bytes: 4096 00000080:00200000:2.0:1789760735.804703:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.804709:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.804711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23519232, 23523328) 0 0 00000080:00200000:2.0:1789760735.804715:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.804726:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5742, 5742] 00020000:00200000:2.0:1789760735.804732:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23519232 chunk: [23519232, 23523328) 23523328, 4096 00020000:00200000:2.0:1789760735.804736:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.804749:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5742, 5742] 00000008:00010000:2.0:1789760735.804762:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.804768:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760735.804776:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23519232, 23523328) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.804787:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23519232=0x166e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.804838:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.804841:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23523328 00000020:00010000:2.0:1789760735.804848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.804859:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.804863:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.804866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.804869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.804876:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.804881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.804896:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760735.805254:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23519232<>23523328 00000100:00100000:3.0:1789760735.805304:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.840472:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973061 gen 3 00000008:00010000:3.0:1789760735.840514:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760735.840520:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.840529:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.840556:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760735.840582:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493716864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760735.876736:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5743, 5743] 00000080:00200000:3.0:1789760735.876749:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760735.876755:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760735.876767:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23527424 i_blocks 47111 00000020:00200000:3.0:1789760735.876770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760735.876780:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760735.876811:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23527424, count: 4096 00000080:00200000:3.0:1789760735.876817:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23527424, bytes: 4096 00000080:00200000:3.0:1789760735.876826:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760735.876834:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760735.876838:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23527424, 23531520) 0 0 00000080:00200000:3.0:1789760735.876843:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760735.876855:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5744, 5744] 00020000:00200000:3.0:1789760735.876862:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23527424 chunk: [23527424, 23531520) 23531520, 4096 00020000:00200000:3.0:1789760735.876867:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760735.876882:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5744, 5744] 00000008:00010000:3.0:1789760735.876896:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760735.876902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760735.876911:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23527424, 23531520) append 0 append lockpos 0 00000080:00200000:3.0:1789760735.876923:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23527424=0x1670000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760735.876971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760735.876974:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23531520 00000020:00010000:3.0:1789760735.876981:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760735.876985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760735.876988:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760735.876991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760735.876994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760735.877000:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760735.877005:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760735.877019:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760735.877098:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 23527424<>23531520 00000100:00100000:2.0:1789760735.877167:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760735.925708:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973063 gen 3 00000008:00010000:3.0:1789760735.925756:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760735.925762:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760735.925769:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760735.925790:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760735.925815:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760736.020230:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5745, 5745] 00000080:00200000:3.0:1789760736.020243:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.020247:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.020260:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23535616 i_blocks 47111 00000020:00200000:3.0:1789760736.020263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.020273:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.020310:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23535616, count: 4096 00000080:00200000:3.0:1789760736.020316:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23535616, bytes: 4096 00000080:00200000:3.0:1789760736.020325:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.020332:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.020334:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23535616, 23539712) 0 0 00000080:00200000:3.0:1789760736.020338:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.020350:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5746, 5746] 00020000:00200000:3.0:1789760736.020356:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23535616 chunk: [23535616, 23539712) 23539712, 4096 00020000:00200000:3.0:1789760736.020359:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.020382:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5746, 5746] 00000008:00010000:3.0:1789760736.020424:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.020429:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.020438:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23535616, 23539712) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.020449:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23535616=0x1672000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.020494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.020497:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23539712 00000020:00010000:3.0:1789760736.020503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.020507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.020510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.020513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.020515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.020521:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.020526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.020541:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760736.020617:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 23535616<>23539712 00000100:00100000:3.0:1789760736.020659:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760736.020672:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717376:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.083304:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973065 gen 3 00000008:00010000:3.0:1789760736.083335:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760736.083341:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.083349:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.083369:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760736.083438:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717376:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.083572:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5746, 5746] 00000080:00200000:2.0:1789760736.083586:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.083591:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.083614:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23539712 i_blocks 47111 00000020:00200000:2.0:1789760736.083618:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.083626:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.083663:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23539712, count: 4096 00000080:00200000:2.0:1789760736.083669:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23539712, bytes: 4096 00000080:00200000:2.0:1789760736.083677:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.083684:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.083686:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23539712, 23543808) 0 0 00000080:00200000:2.0:1789760736.083690:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.083702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5747, 5747] 00020000:00200000:2.0:1789760736.083709:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23539712 chunk: [23539712, 23543808) 23543808, 4096 00020000:00200000:2.0:1789760736.083712:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.083726:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5747, 5747] 00000008:00010000:2.0:1789760736.083739:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.083743:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.083753:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23539712, 23543808) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.083763:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23539712=0x1673000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.083806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.083820:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23543808 00000020:00010000:2.0:1789760736.083827:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.083830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.083833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.083836:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.083839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.083845:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.083851:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.083864:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760736.083936:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23539712<>23543808 00000100:00100000:2.0:1789760736.083969:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760736.083991:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493717504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.151346:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973066 gen 3 00000008:00010000:3.0:1789760736.151408:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760736.151415:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.151426:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.151452:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760736.151479:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493717504:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.151695:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5747, 5747] 00000080:00200000:2.0:1789760736.151709:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.151715:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.151726:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23543808 i_blocks 47111 00000020:00200000:2.0:1789760736.151729:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.151738:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.151763:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23543808, count: 4096 00000080:00200000:2.0:1789760736.151769:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23543808, bytes: 4096 00000080:00200000:2.0:1789760736.151779:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.151786:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.151788:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23543808, 23547904) 0 0 00000080:00200000:2.0:1789760736.151791:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.151806:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5748, 5748] 00020000:00200000:2.0:1789760736.152934:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23543808 chunk: [23543808, 23547904) 23547904, 4096 00020000:00200000:2.0:1789760736.152942:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.153056:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5748, 5748] 00000008:00010000:2.0:1789760736.153082:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.153087:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.153097:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23543808, 23547904) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.153109:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23543808=0x1674000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.153156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.153159:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23547904 00000020:00010000:2.0:1789760736.153165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.153168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.153171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.153174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.153176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.153183:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.153188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.153200:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760736.153278:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23543808<>23547904 00000100:00100000:3.0:1789760736.153715:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493717632:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.210755:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973067 gen 3 00000008:00010000:3.0:1789760736.210793:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760736.210800:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.210822:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.210841:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760736.210871:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493717632:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.211201:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5748, 5748] 00000080:00200000:2.0:1789760736.211213:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.211218:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.211230:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23547904 i_blocks 47111 00000020:00200000:2.0:1789760736.211234:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.211243:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.211269:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23547904, count: 4096 00000080:00200000:2.0:1789760736.211275:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23547904, bytes: 4096 00000080:00200000:2.0:1789760736.211285:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.211292:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.211295:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23547904, 23552000) 0 0 00000080:00200000:2.0:1789760736.211299:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.211312:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5749, 5749] 00020000:00200000:2.0:1789760736.211318:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23547904 chunk: [23547904, 23552000) 23552000, 4096 00020000:00200000:2.0:1789760736.211322:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.211337:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5749, 5749] 00000008:00010000:2.0:1789760736.211349:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.211354:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.211363:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23547904, 23552000) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.211386:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23547904=0x1675000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.211458:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.211461:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23552000 00000020:00010000:2.0:1789760736.211468:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.211472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.211475:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.211478:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.211481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.211487:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.211493:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.211507:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760736.212250:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23547904<>23552000 00000100:00100000:3.0:1789760736.213136:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717760:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.274874:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973068 gen 3 00000008:00010000:3.0:1789760736.274908:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760736.274914:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.274922:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.274941:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760736.274967:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493717760:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.342730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5750, 5750] 00000080:00200000:2.0:1789760736.342744:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.342837:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.342954:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23556096 i_blocks 47111 00000020:00200000:2.0:1789760736.342959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.342969:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.342999:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23556096, count: 4096 00000080:00200000:2.0:1789760736.343006:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23556096, bytes: 4096 00000080:00200000:2.0:1789760736.343018:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.343025:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.343027:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23556096, 23560192) 0 0 00000080:00200000:2.0:1789760736.343032:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.343046:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5751, 5751] 00020000:00200000:2.0:1789760736.343054:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23556096 chunk: [23556096, 23560192) 23560192, 4096 00020000:00200000:2.0:1789760736.343058:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.343074:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5751, 5751] 00000008:00010000:2.0:1789760736.343089:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.343093:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.343169:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23556096, 23560192) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.343181:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23556096=0x1677000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.343226:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.343228:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23560192 00000020:00010000:2.0:1789760736.343234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.343237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.343240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.343244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.343247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.343253:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.343258:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.343272:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760736.344002:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23556096<>23560192 00000100:00100000:2.0:1789760736.344043:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760736.344059:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.404690:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973070 gen 3 00000008:00010000:3.0:1789760736.404728:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760736.404734:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.404743:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.404764:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760736.404792:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718016:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760736.452944:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5752, 5752] 00000080:00200000:3.0:1789760736.452956:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.452963:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.452974:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23564288 i_blocks 47111 00000020:00200000:3.0:1789760736.452977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.452986:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.453014:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23564288, count: 4096 00000080:00200000:3.0:1789760736.453020:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23564288, bytes: 4096 00000080:00200000:3.0:1789760736.453028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.453035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.453036:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23564288, 23568384) 0 0 00000080:00200000:3.0:1789760736.453040:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.453053:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5753, 5753] 00020000:00200000:3.0:1789760736.453059:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23564288 chunk: [23564288, 23568384) 23568384, 4096 00020000:00200000:3.0:1789760736.453062:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.453077:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5753, 5753] 00000008:00010000:3.0:1789760736.453091:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.453096:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.453104:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23564288, 23568384) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.453115:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23564288=0x1679000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.453161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.453164:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23568384 00000020:00010000:3.0:1789760736.453171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.453174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.453177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.453180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.453184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.453190:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.453195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.453210:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760736.453450:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 23564288<>23568384 00000100:00100000:2.0:1789760736.453513:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718272:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760736.492971:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760736.502712:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973071 gen 3 00000100:00100000:3.0:1789760736.502750:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148380 x1876699493718272/t4294973072(4294973072) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760752 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760736.502787:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760736.502793:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.502801:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.502822:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760736.502844:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718272:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760736.553389:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5754, 5754] 00000080:00200000:3.0:1789760736.553439:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.553444:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.553456:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23572480 i_blocks 47111 00000020:00200000:3.0:1789760736.553460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.553468:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.553495:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23572480, count: 4096 00000080:00200000:3.0:1789760736.553501:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23572480, bytes: 4096 00000080:00200000:3.0:1789760736.553510:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.553517:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.553519:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23572480, 23576576) 0 0 00000080:00200000:3.0:1789760736.553523:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.553536:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5755, 5755] 00020000:00200000:3.0:1789760736.553544:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23572480 chunk: [23572480, 23576576) 23576576, 4096 00020000:00200000:3.0:1789760736.553547:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.553562:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5755, 5755] 00000008:00010000:3.0:1789760736.553576:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.553581:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.553589:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23572480, 23576576) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.553599:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23572480=0x167b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.553642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.553645:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23576576 00000020:00010000:3.0:1789760736.553652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.553655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.553658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.553662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.553664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.553670:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.553676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.553692:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760736.553767:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 23572480<>23576576 00000100:00100000:3.0:1789760736.553809:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760736.553823:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.600264:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973073 gen 3 00000100:00100000:3.0:1789760736.600299:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514ad80 x1876699493718528/t4294973074(4294973074) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760752 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760736.600336:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760736.600342:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.600350:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.600380:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760736.600441:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718528:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760736.642041:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5756, 5756] 00000080:00200000:3.0:1789760736.642053:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.642058:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.642069:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23580672 i_blocks 47111 00000020:00200000:3.0:1789760736.642072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.642082:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.642110:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23580672, count: 4096 00000080:00200000:3.0:1789760736.642116:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23580672, bytes: 4096 00000080:00200000:3.0:1789760736.642125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.642132:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.642134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23580672, 23584768) 0 0 00000080:00200000:3.0:1789760736.642138:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.642152:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5757, 5757] 00020000:00200000:3.0:1789760736.642158:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23580672 chunk: [23580672, 23584768) 23584768, 4096 00020000:00200000:3.0:1789760736.642162:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.642178:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5757, 5757] 00000008:00010000:3.0:1789760736.642192:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.642197:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.642207:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23580672, 23584768) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.642218:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23580672=0x167d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.642265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.642268:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23584768 00000020:00010000:3.0:1789760736.642275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.642278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.642281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.642284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.642287:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.642294:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.642300:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.642315:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760736.642438:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 23580672<>23584768 00000100:00100000:2.0:1789760736.642502:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.680492:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973075 gen 3 00000100:00100000:3.0:1789760736.680523:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148000 x1876699493718784/t4294973076(4294973076) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760752 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760736.680558:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760736.680563:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.680571:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.680644:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760736.680667:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493718784:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760736.736618:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5758, 5758] 00000080:00200000:3.0:1789760736.736632:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.736637:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.736651:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23588864 i_blocks 47111 00000020:00200000:3.0:1789760736.736654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.736663:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.736688:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23588864, count: 4096 00000080:00200000:3.0:1789760736.736695:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23588864, bytes: 4096 00000080:00200000:3.0:1789760736.736703:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.736711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.736713:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23588864, 23592960) 0 0 00000080:00200000:3.0:1789760736.736716:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.736728:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5759, 5759] 00020000:00200000:3.0:1789760736.736735:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23588864 chunk: [23588864, 23592960) 23592960, 4096 00020000:00200000:3.0:1789760736.736738:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.736753:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5759, 5759] 00000008:00010000:3.0:1789760736.736766:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.736771:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.736780:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23588864, 23592960) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.736790:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23588864=0x167f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.736837:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.736839:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23592960 00000020:00010000:3.0:1789760736.736855:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.736858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.736861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.736865:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.736867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.736873:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.736878:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.785066:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5759, 5759] 00000080:00200000:3.0:1789760736.785078:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.785083:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.785094:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23592960 i_blocks 47111 00000020:00200000:3.0:1789760736.785097:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.785105:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.785132:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23592960, count: 4096 00000080:00200000:3.0:1789760736.785138:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23592960, bytes: 4096 00000080:00200000:3.0:1789760736.785146:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.785153:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.785155:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23592960, 23597056) 0 0 00000080:00200000:3.0:1789760736.785159:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.785171:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5760, 5760] 00020000:00200000:3.0:1789760736.785178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23592960 chunk: [23592960, 23597056) 23597056, 4096 00020000:00200000:3.0:1789760736.785181:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.785195:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5760, 5760] 00000008:00010000:3.0:1789760736.785209:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.785214:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760736.785223:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23592960, 23597056) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.785235:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23592960=0x1680000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.785290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.785293:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23597056 00000020:00010000:3.0:1789760736.785304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.785307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.785311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.785314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.785316:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.785323:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.785328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.785343:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760736.785477:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23592960<>23597056 00000100:00100000:3.0:1789760736.785520:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760736.785534:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493719168:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760736.813251:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493719424:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760736.813355:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1789760736.813364:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2368:1876699493719296:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760736.813447:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2368:1876699493719552:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760736.819260:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493719424:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760736.820794:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2368:1876699493719296:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760736.822208:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2368:1876699493719552:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760736.826259:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973079 gen 3 00000008:00010000:3.0:1789760736.826294:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760736.826300:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.826307:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.826329:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760736.826354:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493719168:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.826489:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5760, 5760] 00000080:00200000:2.0:1789760736.826501:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.826507:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.826519:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23597056 i_blocks 47111 00000020:00200000:2.0:1789760736.826523:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.826532:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.826562:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23597056, count: 4096 00000080:00200000:2.0:1789760736.826569:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23597056, bytes: 4096 00000080:00200000:2.0:1789760736.826577:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.826583:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.826585:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23597056, 23601152) 0 0 00000080:00200000:2.0:1789760736.826590:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.826603:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5761, 5761] 00020000:00200000:2.0:1789760736.826610:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23597056 chunk: [23597056, 23601152) 23601152, 4096 00020000:00200000:2.0:1789760736.826614:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.826630:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5761, 5761] 00000008:00010000:2.0:1789760736.826643:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.826648:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.826657:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23597056, 23601152) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.826667:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23597056=0x1681000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.826713:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.826716:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23601152 00000020:00010000:2.0:1789760736.826723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.826727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.826730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.826733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.826736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.826742:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.826747:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.826762:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760736.826839:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23597056<>23601152 00000100:00100000:2.0:1789760736.826885:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760736.826899:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719680:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760736.886761:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760736.886772:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.886781:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.886798:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760736.886833:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.887121:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5761, 5761] 00000080:00200000:2.0:1789760736.887133:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.887139:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.887152:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23601152 i_blocks 47111 00000020:00200000:2.0:1789760736.887155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.887164:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.887189:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23601152, count: 4096 00000080:00200000:2.0:1789760736.887195:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23601152, bytes: 4096 00000080:00200000:2.0:1789760736.887203:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.887210:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.887212:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23601152, 23605248) 0 0 00000080:00200000:2.0:1789760736.887216:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.887229:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5762, 5762] 00020000:00200000:2.0:1789760736.887236:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23601152 chunk: [23601152, 23605248) 23605248, 4096 00020000:00200000:2.0:1789760736.887239:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.887255:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5762, 5762] 00000008:00010000:2.0:1789760736.887268:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.887273:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.887282:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23601152, 23605248) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.887292:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23601152=0x1682000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.887335:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.887337:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23605248 00000020:00010000:2.0:1789760736.887344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.887347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.887350:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.887353:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.887356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.887362:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.887381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.887480:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760736.887555:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23601152<>23605248 00000100:00100000:3.0:1789760736.887693:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760736.927055:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973081 gen 3 00000008:00010000:3.0:1789760736.927105:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760736.927112:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.927120:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.927138:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760736.927161:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.927345:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5762, 5762] 00000080:00200000:2.0:1789760736.927358:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.927374:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.927388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23605248 i_blocks 47111 00000020:00200000:2.0:1789760736.927391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.927424:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.927451:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23605248, count: 4096 00000080:00200000:2.0:1789760736.927457:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23605248, bytes: 4096 00000080:00200000:2.0:1789760736.927465:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.927472:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.927474:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23605248, 23609344) 0 0 00000080:00200000:2.0:1789760736.927478:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.927490:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5763, 5763] 00020000:00200000:2.0:1789760736.927496:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23605248 chunk: [23605248, 23609344) 23609344, 4096 00020000:00200000:2.0:1789760736.927500:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.927516:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5763, 5763] 00000008:00010000:2.0:1789760736.927529:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.927534:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.927543:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23605248, 23609344) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.927553:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23605248=0x1683000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.927594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.927596:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23609344 00000020:00010000:2.0:1789760736.927602:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.927606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.927609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.927612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.927616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.927621:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.927627:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.927640:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760736.927717:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23605248<>23609344 00000100:00100000:3.0:1789760736.927813:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760736.927826:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719936:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760736.974280:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760736.974292:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760736.974301:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760736.974321:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760736.974346:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493719936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760736.974561:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5763, 5763] 00000080:00200000:2.0:1789760736.974574:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.974579:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.974590:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23609344 i_blocks 47111 00000020:00200000:2.0:1789760736.974594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.974602:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.974640:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23609344, count: 4096 00000080:00200000:2.0:1789760736.974647:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23609344, bytes: 4096 00000080:00200000:2.0:1789760736.974655:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.974662:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.974664:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23609344, 23613440) 0 0 00000080:00200000:2.0:1789760736.974668:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.974682:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5764, 5764] 00020000:00200000:2.0:1789760736.974689:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23609344 chunk: [23609344, 23613440) 23613440, 4096 00020000:00200000:2.0:1789760736.974693:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.974708:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5764, 5764] 00000008:00010000:2.0:1789760736.974721:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.974726:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760736.974735:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23609344, 23613440) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.974745:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23609344=0x1684000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.974790:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.974792:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23613440 00000020:00010000:2.0:1789760736.974798:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.974801:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.974804:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.974807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.974810:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.974829:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.974835:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.974851:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760736.975058:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23609344<>23613440 00000100:00100000:3.0:1789760736.975117:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.038582:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973083 gen 3 00000008:00010000:3.0:1789760737.038628:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760737.038635:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.038644:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.038663:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760737.038689:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.038781:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5764, 5764] 00000080:00200000:2.0:1789760737.038794:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.038800:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.038812:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23613440 i_blocks 47111 00000020:00200000:2.0:1789760737.038815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.038824:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.038862:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23613440, count: 4096 00000080:00200000:2.0:1789760737.038869:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23613440, bytes: 4096 00000080:00200000:2.0:1789760737.038876:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.038883:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.038885:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23613440, 23617536) 0 0 00000080:00200000:2.0:1789760737.038888:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.038901:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5765, 5765] 00020000:00200000:2.0:1789760737.038908:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23613440 chunk: [23613440, 23617536) 23617536, 4096 00020000:00200000:2.0:1789760737.038911:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.038927:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5765, 5765] 00000008:00010000:2.0:1789760737.038940:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.038945:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.038955:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23613440, 23617536) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.038967:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23613440=0x1685000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.039012:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.039015:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23617536 00000020:00010000:2.0:1789760737.039021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.039024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.039027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.039030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.039032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.039038:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.039043:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.039058:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760737.039135:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23613440<>23617536 00000100:00100000:3.0:1789760737.039332:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.039346:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.119861:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973084 gen 3 00000008:00010000:3.0:1789760737.119896:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760737.119902:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.119911:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.119932:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760737.119958:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.120060:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5765, 5765] 00000080:00200000:2.0:1789760737.120073:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.120078:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.120091:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23617536 i_blocks 47111 00000020:00200000:2.0:1789760737.120094:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.120104:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.120132:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23617536, count: 4096 00000080:00200000:2.0:1789760737.120138:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23617536, bytes: 4096 00000080:00200000:2.0:1789760737.120256:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.120265:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.120267:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23617536, 23621632) 0 0 00000080:00200000:2.0:1789760737.120271:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.120285:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5766, 5766] 00020000:00200000:2.0:1789760737.120825:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23617536 chunk: [23617536, 23621632) 23621632, 4096 00020000:00200000:2.0:1789760737.120830:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.120848:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5766, 5766] 00000008:00010000:2.0:1789760737.120863:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.120867:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.120876:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23617536, 23621632) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.120887:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23617536=0x1686000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.120932:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.120935:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23621632 00000020:00010000:2.0:1789760737.120941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.120944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.120947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.120949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.120952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.120958:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.120963:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.120980:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760737.121054:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23617536<>23621632 00000100:00100000:3.0:1789760737.121129:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720320:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760737.202344:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760737.202355:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.202373:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.202408:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760737.202435:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720320:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.202977:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5766, 5766] 00000080:00200000:2.0:1789760737.202991:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.202997:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.203008:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23621632 i_blocks 47111 00000020:00200000:2.0:1789760737.203011:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.203021:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.203048:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23621632, count: 4096 00000080:00200000:2.0:1789760737.203053:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23621632, bytes: 4096 00000080:00200000:2.0:1789760737.203064:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.203071:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.203074:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23621632, 23625728) 0 0 00000080:00200000:2.0:1789760737.203077:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.203090:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5767, 5767] 00020000:00200000:2.0:1789760737.203096:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23621632 chunk: [23621632, 23625728) 23625728, 4096 00020000:00200000:2.0:1789760737.203100:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.203116:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5767, 5767] 00000008:00010000:2.0:1789760737.203130:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.203134:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.203144:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23621632, 23625728) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.203154:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23621632=0x1687000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.203196:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.203199:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23625728 00000020:00010000:2.0:1789760737.203205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.203208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.203211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.203214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.203217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.203222:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.203228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.203243:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760737.203320:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23621632<>23625728 00000100:00100000:3.0:1789760737.203439:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493720448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.267016:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973086 gen 3 00000008:00010000:3.0:1789760737.267061:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760737.267069:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.267077:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.267096:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760737.267121:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493720448:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.267212:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5767, 5767] 00000080:00200000:2.0:1789760737.267225:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.267231:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.267243:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23625728 i_blocks 47111 00000020:00200000:2.0:1789760737.267246:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.267255:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.267281:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23625728, count: 4096 00000080:00200000:2.0:1789760737.267288:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23625728, bytes: 4096 00000080:00200000:2.0:1789760737.267295:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.267302:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.267304:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23625728, 23629824) 0 0 00000080:00200000:2.0:1789760737.267308:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.267320:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5768, 5768] 00020000:00200000:2.0:1789760737.267327:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23625728 chunk: [23625728, 23629824) 23629824, 4096 00020000:00200000:2.0:1789760737.267330:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.267345:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5768, 5768] 00000008:00010000:2.0:1789760737.267359:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.267373:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.267383:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23625728, 23629824) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.267425:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23625728=0x1688000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.267482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.267486:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23629824 00000020:00010000:2.0:1789760737.267493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.267497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.267500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.267503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.267505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.267511:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.267517:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.267531:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760737.267731:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23625728<>23629824 00000100:00100000:3.0:1789760737.267875:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.365131:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973087 gen 3 00000008:00010000:3.0:1789760737.365169:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760737.365176:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.365186:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.365205:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760737.365229:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720576:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.365346:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5768, 5768] 00000080:00200000:2.0:1789760737.365359:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.365374:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.365385:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23629824 i_blocks 47111 00000020:00200000:2.0:1789760737.365389:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.365419:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.365445:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23629824, count: 4096 00000080:00200000:2.0:1789760737.365464:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23629824, bytes: 4096 00000080:00200000:2.0:1789760737.365475:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.365482:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.365485:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23629824, 23633920) 0 0 00000080:00200000:2.0:1789760737.365489:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.365502:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5769, 5769] 00020000:00200000:2.0:1789760737.365510:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23629824 chunk: [23629824, 23633920) 23633920, 4096 00020000:00200000:2.0:1789760737.365513:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.365529:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5769, 5769] 00000008:00010000:2.0:1789760737.365542:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.365547:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.365556:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23629824, 23633920) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.365567:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23629824=0x1689000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.365611:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.365614:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23633920 00000020:00010000:2.0:1789760737.365621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.365625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.365627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.365630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.365633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.365640:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.365645:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.365657:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760737.365733:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23629824<>23633920 00000100:00100000:3.0:1789760737.365948:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.365964:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720704:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.424519:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973088 gen 3 00000008:00010000:3.0:1789760737.424558:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760737.424565:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.424575:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.424594:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760737.424618:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720704:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.424714:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5769, 5769] 00000080:00200000:2.0:1789760737.424727:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.424732:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.424743:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23633920 i_blocks 47111 00000020:00200000:2.0:1789760737.424746:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.424754:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.424780:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23633920, count: 4096 00000080:00200000:2.0:1789760737.424785:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23633920, bytes: 4096 00000080:00200000:2.0:1789760737.424794:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.424801:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.424803:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23633920, 23638016) 0 0 00000080:00200000:2.0:1789760737.424807:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.424940:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5770, 5770] 00020000:00200000:2.0:1789760737.424947:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23633920 chunk: [23633920, 23638016) 23638016, 4096 00020000:00200000:2.0:1789760737.424951:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.424969:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5770, 5770] 00000008:00010000:2.0:1789760737.424982:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.424988:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.424997:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23633920, 23638016) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.425008:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23633920=0x168a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.425053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.425055:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23638016 00000020:00010000:2.0:1789760737.425061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.425064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.425068:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.425070:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.425073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.425079:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.425085:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.425099:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760737.425176:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23633920<>23638016 00000100:00100000:3.0:1789760737.425236:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720832:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.480905:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973089 gen 3 00000008:00010000:3.0:1789760737.480943:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760737.480950:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.480959:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.480978:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760737.481005:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.481157:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5770, 5770] 00000080:00200000:2.0:1789760737.481171:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.481176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.481190:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23638016 i_blocks 47111 00000020:00200000:2.0:1789760737.481193:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.481203:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.481231:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23638016, count: 4096 00000080:00200000:2.0:1789760737.481237:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23638016, bytes: 4096 00000080:00200000:2.0:1789760737.481244:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.481250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.481252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23638016, 23642112) 0 0 00000080:00200000:2.0:1789760737.481255:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.481267:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5771, 5771] 00020000:00200000:2.0:1789760737.481273:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23638016 chunk: [23638016, 23642112) 23642112, 4096 00020000:00200000:2.0:1789760737.481277:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.481302:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5771, 5771] 00000008:00010000:2.0:1789760737.481317:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.481321:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.481330:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23638016, 23642112) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.481341:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23638016=0x168b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.481440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.481443:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23642112 00000020:00010000:2.0:1789760737.481458:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.481461:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.481464:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.481467:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.481469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.481475:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.481480:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.481495:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760737.481570:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23638016<>23642112 00000100:00100000:3.0:1789760737.482209:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.482229:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720960:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.538441:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973090 gen 3 00000008:00010000:3.0:1789760737.538486:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760737.538492:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.538502:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.538517:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760737.538541:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493720960:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.538730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5771, 5771] 00000080:00200000:2.0:1789760737.538742:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.538747:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.538759:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23642112 i_blocks 47111 00000020:00200000:2.0:1789760737.538763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.538771:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.538799:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23642112, count: 4096 00000080:00200000:2.0:1789760737.538805:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23642112, bytes: 4096 00000080:00200000:2.0:1789760737.538813:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.538820:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.538822:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23642112, 23646208) 0 0 00000080:00200000:2.0:1789760737.538825:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.538836:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5772, 5772] 00020000:00200000:2.0:1789760737.538856:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23642112 chunk: [23642112, 23646208) 23646208, 4096 00020000:00200000:2.0:1789760737.538860:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.538875:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5772, 5772] 00000008:00010000:2.0:1789760737.538888:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.538893:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.538901:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23642112, 23646208) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.538912:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23642112=0x168c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.538955:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.538958:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23646208 00000020:00010000:2.0:1789760737.538964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.538967:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.538970:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.538973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.538975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.538980:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.538986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.539000:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760737.539072:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23642112<>23646208 00000100:00100000:3.0:1789760737.539206:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760737.539220:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721088:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.588609:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973091 gen 3 00000008:00010000:3.0:1789760737.588644:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760737.588650:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.588660:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.588679:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760737.588715:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721088:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.588880:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5772, 5772] 00000080:00200000:2.0:1789760737.588892:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.588898:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.588909:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23646208 i_blocks 47111 00000020:00200000:2.0:1789760737.588912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.588921:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.588948:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23646208, count: 4096 00000080:00200000:2.0:1789760737.588953:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23646208, bytes: 4096 00000080:00200000:2.0:1789760737.588961:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.588968:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.588969:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23646208, 23650304) 0 0 00000080:00200000:2.0:1789760737.588973:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.588985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5773, 5773] 00020000:00200000:2.0:1789760737.588992:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23646208 chunk: [23646208, 23650304) 23650304, 4096 00020000:00200000:2.0:1789760737.588996:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.589010:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5773, 5773] 00000008:00010000:2.0:1789760737.589023:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.589028:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.589038:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23646208, 23650304) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.589049:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23646208=0x168d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.589105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.589108:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23650304 00000020:00010000:2.0:1789760737.589114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.589118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.589121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.589124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.589127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.589134:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.589139:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.589154:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760737.589229:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23646208<>23650304 00000100:00100000:3.0:1789760737.589290:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.631006:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973092 gen 3 00000008:00010000:3.0:1789760737.631046:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760737.631053:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.631062:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.631136:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760737.631167:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721216:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.631806:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5773, 5773] 00000080:00200000:2.0:1789760737.631819:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.631824:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.631838:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23650304 i_blocks 47111 00000020:00200000:2.0:1789760737.633757:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.633780:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.633810:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23650304, count: 4096 00000080:00200000:2.0:1789760737.633819:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23650304, bytes: 4096 00000080:00200000:2.0:1789760737.633830:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.633837:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.634033:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23650304, 23654400) 0 0 00000080:00200000:2.0:1789760737.634039:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.634053:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5774, 5774] 00020000:00200000:2.0:1789760737.634060:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23650304 chunk: [23650304, 23654400) 23654400, 4096 00020000:00200000:2.0:1789760737.634064:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.634082:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5774, 5774] 00000008:00010000:2.0:1789760737.634096:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.634101:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.634110:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23650304, 23654400) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.634121:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23650304=0x168e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.634179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.634182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23654400 00000020:00010000:2.0:1789760737.634189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.634192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.634195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.634198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.634201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.634208:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.634214:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.634240:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760737.634319:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23650304<>23654400 00000100:00100000:3.0:1789760737.634425:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493721344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.688626:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973093 gen 3 00000008:00010000:3.0:1789760737.688772:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760737.688779:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.688789:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.688858:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760737.688883:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493721344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.695533:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5774, 5774] 00000080:00200000:2.0:1789760737.695551:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.695557:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.695570:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23654400 i_blocks 47111 00000020:00200000:2.0:1789760737.695574:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.695586:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.695612:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23654400, count: 4096 00000080:00200000:2.0:1789760737.695618:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23654400, bytes: 4096 00000080:00200000:2.0:1789760737.695629:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.695635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.695637:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23654400, 23658496) 0 0 00000080:00200000:2.0:1789760737.695640:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.697503:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5775, 5775] 00020000:00200000:2.0:1789760737.697516:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23654400 chunk: [23654400, 23658496) 23658496, 4096 00020000:00200000:2.0:1789760737.697520:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.697542:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5775, 5775] 00000008:00010000:2.0:1789760737.697558:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.697562:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.697573:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23654400, 23658496) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.697585:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23654400=0x168f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.697636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.697638:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23658496 00000020:00010000:2.0:1789760737.697644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.697647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.697649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.697662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.697665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.697673:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.697678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.697698:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760737.697777:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23654400<>23658496 00000100:00100000:3.0:1789760737.698009:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760737.698027:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493721472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.738699:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973094 gen 3 00000008:00010000:3.0:1789760737.738738:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760737.738744:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.738754:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.738777:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760737.738802:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493721472:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.739013:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5775, 5775] 00000080:00200000:2.0:1789760737.739035:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.739040:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.739052:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23658496 i_blocks 47111 00000020:00200000:2.0:1789760737.739056:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.739065:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.739272:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23658496, count: 4096 00000080:00200000:2.0:1789760737.739280:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23658496, bytes: 4096 00000080:00200000:2.0:1789760737.739289:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.739296:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.739299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23658496, 23662592) 0 0 00000080:00200000:2.0:1789760737.739303:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.739314:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5776, 5776] 00020000:00200000:2.0:1789760737.739321:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23658496 chunk: [23658496, 23662592) 23662592, 4096 00020000:00200000:2.0:1789760737.739325:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.739340:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5776, 5776] 00000008:00010000:2.0:1789760737.739353:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.739368:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.739377:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23658496, 23662592) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.739387:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23658496=0x1690000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.739477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.739480:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23662592 00000020:00010000:2.0:1789760737.739486:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.739489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.739492:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.739495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.739498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.739504:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.739511:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.739527:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760737.739606:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23658496<>23662592 00000100:00100000:3.0:1789760737.740775:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760737.740799:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721600:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760737.796991:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760737.797002:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.797012:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.797031:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760737.797057:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760737.797957:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5776, 5776] 00000080:00200000:2.0:1789760737.797974:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.797979:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.797991:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23662592 i_blocks 47111 00000020:00200000:2.0:1789760737.797995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.798006:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.798033:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23662592, count: 4096 00000080:00200000:2.0:1789760737.798039:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23662592, bytes: 4096 00000080:00200000:2.0:1789760737.798049:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.798056:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.798058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23662592, 23666688) 0 0 00000080:00200000:2.0:1789760737.798062:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.798075:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5777, 5777] 00020000:00200000:2.0:1789760737.798082:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23662592 chunk: [23662592, 23666688) 23666688, 4096 00020000:00200000:2.0:1789760737.798085:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.798100:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5777, 5777] 00000008:00010000:2.0:1789760737.798114:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.798118:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760737.798128:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23662592, 23666688) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.798140:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23662592=0x1691000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.798187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.798190:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23666688 00000020:00010000:2.0:1789760737.798780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.798784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.798786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.798790:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.798793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.798800:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.798806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.798826:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760737.798972:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23662592<>23666688 00000100:00100000:3.0:1789760737.799046:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721728:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760737.828761:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973095 gen 3 00000100:00100000:3.0:1789760737.828791:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80a80 x1876699493721728/t4294973096(4294973096) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760753 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760737.828830:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760737.828837:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760737.828898:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760737.828917:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760737.828943:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493721728:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760737.830210:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493721856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760737.906661:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973097 gen 3 00000008:00010000:1.0:1789760737.906709:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760737.906715:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.906724:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.906753:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760737.906779:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493721856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760737.907917:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760737.907934:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493721984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760737.935460:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973098 gen 3 00000008:00010000:1.0:1789760737.935497:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760737.935503:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.935512:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.935522:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760737.935548:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493721984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760737.936446:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722112:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.015453:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973099 gen 3 00000008:00010000:1.0:1789760738.015487:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760738.015494:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.015503:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.015518:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760738.015543:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722112:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.017315:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493722240:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760738.063987:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760738.063998:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.064007:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.064077:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760738.064105:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493722240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.064835:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.102630:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973100 gen 3 00000100:00100000:1.0:1789760738.102657:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ce00 x1876699493722368/t4294973101(4294973101) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760738.102692:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760738.102697:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.102705:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.102724:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760738.102745:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.104272:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.104530:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760738.179695:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973101 gen 3 00000100:00100000:1.0:1789760738.179729:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304f100 x1876699493722496/t4294973102(4294973102) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760738.179773:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760738.179779:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.179786:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.179805:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760738.179828:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493722496:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760738.180022:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5783, 5783] 00000080:00200000:3.0:1789760738.180035:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.180040:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.180050:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23691264 i_blocks 47111 00000020:00200000:3.0:1789760738.180053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.180062:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.180093:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23691264, count: 4096 00000080:00200000:3.0:1789760738.180099:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23691264, bytes: 4096 00000080:00200000:3.0:1789760738.180108:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.180115:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.180117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23691264, 23695360) 0 0 00000080:00200000:3.0:1789760738.180121:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.180135:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5784, 5784] 00020000:00200000:3.0:1789760738.180143:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23691264 chunk: [23691264, 23695360) 23695360, 4096 00020000:00200000:3.0:1789760738.180147:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.180163:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5784, 5784] 00000008:00010000:3.0:1789760738.180260:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.180267:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760738.180280:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23691264, 23695360) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.180291:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23691264=0x1698000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.180340:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.180342:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23695360 00000020:00010000:3.0:1789760738.180349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.180352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.180365:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.180369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.180372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.180379:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.180385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.183691:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760738.183798:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23691264<>23695360 00000100:00100000:3.0:1789760738.183856:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760738.183877:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493722624:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.254611:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973102 gen 3 00000100:00100000:3.0:1789760738.254641:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149180 x1876699493722624/t4294973103(4294973103) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760738.254677:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760738.254683:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.254689:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.254707:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760738.254726:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493722624:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.254806:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5784, 5784] 00000080:00200000:2.0:1789760738.254827:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.254832:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.254843:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23695360 i_blocks 47111 00000020:00200000:2.0:1789760738.254846:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.254853:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.255047:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23695360, count: 4096 00000080:00200000:2.0:1789760738.255055:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23695360, bytes: 4096 00000080:00200000:2.0:1789760738.255063:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.255069:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.255071:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23695360, 23699456) 0 0 00000080:00200000:2.0:1789760738.255075:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.255087:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5785, 5785] 00020000:00200000:2.0:1789760738.255093:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23695360 chunk: [23695360, 23699456) 23699456, 4096 00020000:00200000:2.0:1789760738.255096:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.255111:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5785, 5785] 00000008:00010000:2.0:1789760738.255123:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.255128:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.255136:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23695360, 23699456) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.255146:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23695360=0x1699000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.255199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.255202:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23699456 00000020:00010000:2.0:1789760738.255208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.255211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.255214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.255217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.255220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.255226:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.255231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.255245:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760738.255321:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23695360<>23699456 00000100:00100000:3.0:1789760738.255390:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493722752:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.302075:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973103 gen 3 00000100:00100000:3.0:1789760738.302115:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81500 x1876699493722752/t4294973104(4294973104) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760738.302158:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760738.302163:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.302172:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.302201:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760738.302224:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493722752:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.302294:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5785, 5785] 00000080:00200000:2.0:1789760738.302307:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.302312:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.302328:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23699456 i_blocks 47111 00000020:00200000:2.0:1789760738.302332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.302343:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.302388:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23699456, count: 4096 00000080:00200000:2.0:1789760738.302428:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23699456, bytes: 4096 00000080:00200000:2.0:1789760738.302438:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.302445:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.302448:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23699456, 23703552) 0 0 00000080:00200000:2.0:1789760738.302452:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.302466:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5786, 5786] 00020000:00200000:2.0:1789760738.302474:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23699456 chunk: [23699456, 23703552) 23703552, 4096 00020000:00200000:2.0:1789760738.302477:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.302493:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5786, 5786] 00000008:00010000:2.0:1789760738.302506:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.302512:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.302520:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23699456, 23703552) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.302530:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23699456=0x169a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.302578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.302581:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23703552 00000020:00010000:2.0:1789760738.302588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.302592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.302595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.302598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.302601:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.302609:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.302615:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.302632:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760738.302731:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23699456<>23703552 00000100:00100000:3.0:1789760738.302916:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493722880:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760738.350779:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760738.350794:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.350818:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760738.353112:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973105 gen 3 00000008:00010000:3.0:1789760738.353157:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760738.353163:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.353172:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.353191:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760738.353216:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493722880:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.353302:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5786, 5786] 00000080:00200000:2.0:1789760738.353316:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.353322:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.353335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23703552 i_blocks 47111 00000020:00200000:2.0:1789760738.353339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.353349:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.353390:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23703552, count: 4096 00000080:00200000:2.0:1789760738.353431:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23703552, bytes: 4096 00000080:00200000:2.0:1789760738.353439:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.353446:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.353448:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23703552, 23707648) 0 0 00000080:00200000:2.0:1789760738.353452:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.353466:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5787, 5787] 00020000:00200000:2.0:1789760738.353473:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23703552 chunk: [23703552, 23707648) 23707648, 4096 00020000:00200000:2.0:1789760738.353476:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.353492:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5787, 5787] 00000008:00010000:2.0:1789760738.353505:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.353509:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.353518:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23703552, 23707648) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.353529:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23703552=0x169b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.353570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.353573:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23707648 00000020:00010000:2.0:1789760738.353579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.353582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.353585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.353588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.353591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.353597:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.353603:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.353617:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760738.353695:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23703552<>23707648 00000100:00100000:2.0:1789760738.353736:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760738.353749:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723008:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.390894:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973106 gen 3 00000008:00010000:3.0:1789760738.390931:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760738.390937:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.390946:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.390965:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760738.390989:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723008:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.391153:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5787, 5787] 00000080:00200000:2.0:1789760738.391166:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.391172:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.391185:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23707648 i_blocks 47111 00000020:00200000:2.0:1789760738.391188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.391198:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.391228:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23707648, count: 4096 00000080:00200000:2.0:1789760738.391233:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23707648, bytes: 4096 00000080:00200000:2.0:1789760738.391242:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.391250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.391252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23707648, 23711744) 0 0 00000080:00200000:2.0:1789760738.391256:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.391269:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5788, 5788] 00020000:00200000:2.0:1789760738.391276:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23707648 chunk: [23707648, 23711744) 23711744, 4096 00020000:00200000:2.0:1789760738.391280:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.391296:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5788, 5788] 00000008:00010000:2.0:1789760738.391308:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.391313:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.391323:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23707648, 23711744) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.391333:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23707648=0x169c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.391391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.391762:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23711744 00000020:00010000:2.0:1789760738.391770:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.391773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.391776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.391779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.391782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.391788:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.391794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.391812:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760738.392016:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23707648<>23711744 00000100:00100000:3.0:1789760738.392084:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723136:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760738.439635:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760738.439644:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.439652:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.439665:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000080:00200000:2.0:1789760738.439689:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5788, 5788] 00000100:00100000:3.0:1789760738.439689:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723136:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.439702:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.439707:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.439718:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23711744 i_blocks 47111 00000020:00200000:2.0:1789760738.439722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.439730:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.439757:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23711744, count: 4096 00000080:00200000:2.0:1789760738.439763:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23711744, bytes: 4096 00000080:00200000:2.0:1789760738.439771:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.439778:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.439780:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23711744, 23715840) 0 0 00000080:00200000:2.0:1789760738.439783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.439796:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5789, 5789] 00020000:00200000:2.0:1789760738.439802:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23711744 chunk: [23711744, 23715840) 23715840, 4096 00020000:00200000:2.0:1789760738.439806:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.439890:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5789, 5789] 00000008:00010000:2.0:1789760738.439907:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.439912:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.439921:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23711744, 23715840) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.439931:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23711744=0x169d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.439976:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.439978:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23715840 00000020:00010000:2.0:1789760738.439985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.439989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.439991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.439995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.439997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.440004:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.440009:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.440024:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760738.440104:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23711744<>23715840 00000100:00100000:3.0:1789760738.440459:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760738.440473:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723264:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.491099:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973107 gen 3 00000100:00100000:3.0:1789760738.491138:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81f80 x1876699493723264/t4294973108(4294973108) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760738.491176:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760738.491182:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.491190:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.491213:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760738.491243:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723264:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.491371:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5789, 5789] 00000080:00200000:2.0:1789760738.491384:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.491389:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.491502:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23715840 i_blocks 47111 00000020:00200000:2.0:1789760738.491506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.491517:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.491613:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23715840, count: 4096 00000080:00200000:2.0:1789760738.491621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23715840, bytes: 4096 00000080:00200000:2.0:1789760738.491629:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.491637:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.491639:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23715840, 23719936) 0 0 00000080:00200000:2.0:1789760738.491643:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.491655:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5790, 5790] 00020000:00200000:2.0:1789760738.491661:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23715840 chunk: [23715840, 23719936) 23719936, 4096 00020000:00200000:2.0:1789760738.491664:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.491687:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5790, 5790] 00000008:00010000:2.0:1789760738.491698:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.491703:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.491719:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23715840, 23719936) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.491730:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23715840=0x169e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.491781:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.491784:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23719936 00000020:00010000:2.0:1789760738.491791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.491794:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.491797:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.491800:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.491803:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.491809:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.491815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.491947:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760738.492028:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23715840<>23719936 00000100:00100000:3.0:1789760738.492153:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723392:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.547624:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973108 gen 3 00000100:00100000:3.0:1789760738.547656:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83b80 x1876699493723392/t4294973109(4294973109) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760754 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760738.547691:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760738.547696:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.547704:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.547721:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760738.547743:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723392:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.547924:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5790, 5790] 00000080:00200000:2.0:1789760738.547936:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.547941:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.547952:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23719936 i_blocks 47111 00000020:00200000:2.0:1789760738.547955:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.547963:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.547989:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23719936, count: 4096 00000080:00200000:2.0:1789760738.547994:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23719936, bytes: 4096 00000080:00200000:2.0:1789760738.548002:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.548008:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.548010:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23719936, 23724032) 0 0 00000080:00200000:2.0:1789760738.548014:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.548026:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5791, 5791] 00020000:00200000:2.0:1789760738.548031:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23719936 chunk: [23719936, 23724032) 23724032, 4096 00020000:00200000:2.0:1789760738.548035:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.548048:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5791, 5791] 00000008:00010000:2.0:1789760738.548058:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.548063:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.548071:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23719936, 23724032) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.548080:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23719936=0x169f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.548117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.548120:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23724032 00000020:00010000:2.0:1789760738.548125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.548128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.548131:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.548134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.548136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.548142:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.548147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.548159:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760738.548300:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23719936<>23724032 00000100:00100000:3.0:1789760738.548349:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493723520:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.597964:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973110 gen 3 00000008:00010000:3.0:1789760738.598011:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760738.598018:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.598027:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.598050:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760738.598079:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493723520:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.598111:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5791, 5791] 00000080:00200000:2.0:1789760738.598123:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.598128:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.598138:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23724032 i_blocks 47111 00000020:00200000:2.0:1789760738.598141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.598150:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.598179:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23724032, count: 4096 00000080:00200000:2.0:1789760738.598184:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23724032, bytes: 4096 00000080:00200000:2.0:1789760738.598195:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.598203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.598205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23724032, 23728128) 0 0 00000080:00200000:2.0:1789760738.598208:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.598221:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5792, 5792] 00020000:00200000:2.0:1789760738.598229:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23724032 chunk: [23724032, 23728128) 23728128, 4096 00020000:00200000:2.0:1789760738.598233:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.598247:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5792, 5792] 00000008:00010000:2.0:1789760738.598260:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.598264:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.598273:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23724032, 23728128) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.598283:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23724032=0x16a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.598325:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.598328:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23728128 00000020:00010000:2.0:1789760738.598334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.598337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.598340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.598344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.598346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.598363:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.598369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.598384:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760738.598497:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23724032<>23728128 00000100:00100000:3.0:1789760738.598647:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723648:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.640205:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973111 gen 3 00000008:00010000:3.0:1789760738.640260:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760738.640270:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.640283:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.640375:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760738.640523:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723648:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.641096:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5792, 5792] 00000080:00200000:2.0:1789760738.641114:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.641121:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.641137:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23728128 i_blocks 47111 00000020:00200000:2.0:1789760738.641142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.641153:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.641190:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23728128, count: 4096 00000080:00200000:2.0:1789760738.641197:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23728128, bytes: 4096 00000080:00200000:2.0:1789760738.641208:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.641218:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.641220:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23728128, 23732224) 0 0 00000080:00200000:2.0:1789760738.641225:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.641244:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5793, 5793] 00020000:00200000:2.0:1789760738.641253:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23728128 chunk: [23728128, 23732224) 23732224, 4096 00020000:00200000:2.0:1789760738.641258:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.641279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5793, 5793] 00000008:00010000:2.0:1789760738.641664:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.641671:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.641684:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23728128, 23732224) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.641754:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23728128=0x16a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.642117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.642121:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23732224 00000020:00010000:2.0:1789760738.642132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.642136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.642139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.642144:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.642148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.642158:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.642165:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.642188:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760738.642681:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23728128<>23732224 00000100:00100000:3.0:1789760738.643194:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760738.643226:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723776:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.680788:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973112 gen 3 00000008:00010000:3.0:1789760738.680945:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760738.680952:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.680963:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.680978:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000080:00200000:2.0:1789760738.681005:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5793, 5793] 00000100:00100000:3.0:1789760738.681014:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493723776:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.681023:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.681029:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.681045:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23732224 i_blocks 47111 00000020:00200000:2.0:1789760738.681048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.681058:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.681092:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23732224, count: 4096 00000080:00200000:2.0:1789760738.681100:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23732224, bytes: 4096 00000080:00200000:2.0:1789760738.681168:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.681214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.681217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23732224, 23736320) 0 0 00000080:00200000:2.0:1789760738.681222:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.681239:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5794, 5794] 00020000:00200000:2.0:1789760738.681247:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23732224 chunk: [23732224, 23736320) 23736320, 4096 00020000:00200000:2.0:1789760738.681252:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.681274:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5794, 5794] 00000008:00010000:2.0:1789760738.681295:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.681301:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.681321:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23732224, 23736320) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.681337:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23732224=0x16a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.681457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.681460:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23736320 00000020:00010000:2.0:1789760738.681468:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.681472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.681475:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.681479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.681482:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.681492:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.681500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.681525:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760738.681632:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23732224<>23736320 00000100:00100000:2.0:1789760738.681684:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760738.681701:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493723904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.719289:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973113 gen 3 00000008:00010000:3.0:1789760738.719359:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760738.719368:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.719382:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.721108:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760738.721157:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493723904:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.721667:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5794, 5794] 00000080:00200000:2.0:1789760738.721695:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.721701:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.721719:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23736320 i_blocks 47111 00000020:00200000:2.0:1789760738.721725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.721737:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.721770:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23736320, count: 4096 00000080:00200000:2.0:1789760738.721777:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23736320, bytes: 4096 00000080:00200000:2.0:1789760738.721791:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.721800:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.721803:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23736320, 23740416) 0 0 00000080:00200000:2.0:1789760738.721807:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.723056:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5795, 5795] 00020000:00200000:2.0:1789760738.723066:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23736320 chunk: [23736320, 23740416) 23740416, 4096 00020000:00200000:2.0:1789760738.723070:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.723091:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5795, 5795] 00000008:00010000:2.0:1789760738.723108:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.723113:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.723125:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23736320, 23740416) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.723136:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23736320=0x16a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.723191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.723193:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23740416 00000020:00010000:2.0:1789760738.723199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.723202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.723204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.723207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.723209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.723217:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.723223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.723243:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760738.723349:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23736320<>23740416 00000100:00100000:3.0:1789760738.723984:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760738.724013:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.790945:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973114 gen 3 00000008:00010000:3.0:1789760738.790999:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760738.791009:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.791021:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.791122:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760738.791152:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.795659:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5795, 5795] 00000080:00200000:2.0:1789760738.795958:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.795967:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.795985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23740416 i_blocks 47111 00000020:00200000:2.0:1789760738.795991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.796009:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.796045:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23740416, count: 4096 00000080:00200000:2.0:1789760738.796054:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23740416, bytes: 4096 00000080:00200000:2.0:1789760738.796068:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.796078:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.796080:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23740416, 23744512) 0 0 00000080:00200000:2.0:1789760738.796085:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.796102:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5796, 5796] 00020000:00200000:2.0:1789760738.796111:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23740416 chunk: [23740416, 23744512) 23744512, 4096 00020000:00200000:2.0:1789760738.796116:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.796140:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5796, 5796] 00000008:00010000:2.0:1789760738.796159:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.796166:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.796180:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23740416, 23744512) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.796196:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23740416=0x16a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.798843:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.798856:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23744512 00000020:00010000:2.0:1789760738.798872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.798877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.798882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.798886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.799126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.799145:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.799155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.799186:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760738.799311:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23740416<>23744512 00000100:00100000:3.0:1789760738.800006:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760738.800038:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.830066:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973115 gen 3 00000008:00010000:3.0:1789760738.830117:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760738.830126:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.830138:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.830159:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760738.830189:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.830454:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5796, 5796] 00000080:00200000:2.0:1789760738.830470:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.830476:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.830493:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23744512 i_blocks 47111 00000020:00200000:2.0:1789760738.830496:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.830507:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.830550:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23744512, count: 4096 00000080:00200000:2.0:1789760738.830558:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23744512, bytes: 4096 00000080:00200000:2.0:1789760738.830569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.830577:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.830580:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23744512, 23748608) 0 0 00000080:00200000:2.0:1789760738.830585:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.830601:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5797, 5797] 00020000:00200000:2.0:1789760738.830609:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23744512 chunk: [23744512, 23748608) 23748608, 4096 00020000:00200000:2.0:1789760738.830615:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.830634:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5797, 5797] 00000008:00010000:2.0:1789760738.830652:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.830658:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.830672:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23744512, 23748608) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.830686:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23744512=0x16a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.830755:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.830758:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23748608 00000020:00010000:2.0:1789760738.830766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.830769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.830773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.830776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.830779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.830787:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.830794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.830809:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760738.830926:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23744512<>23748608 00000100:00100000:3.0:1789760738.831135:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760738.831155:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724288:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760738.867260:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760738.867273:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.867285:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.867343:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000080:00200000:2.0:1789760738.867348:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5797, 5797] 00000080:00200000:2.0:1789760738.867378:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760738.867386:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.867390:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.867442:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23748608 i_blocks 47111 00000020:00200000:2.0:1789760738.867446:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.867456:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.867489:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23748608, count: 4096 00000080:00200000:2.0:1789760738.867497:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23748608, bytes: 4096 00000080:00200000:2.0:1789760738.867507:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.867516:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.867519:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23748608, 23752704) 0 0 00000080:00200000:2.0:1789760738.867524:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.867540:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5798, 5798] 00020000:00200000:2.0:1789760738.867549:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23748608 chunk: [23748608, 23752704) 23752704, 4096 00020000:00200000:2.0:1789760738.867553:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.867572:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5798, 5798] 00000008:00010000:2.0:1789760738.867589:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.867595:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.867606:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23748608, 23752704) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.867619:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23748608=0x16a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.867673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.867676:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23752704 00000020:00010000:2.0:1789760738.867684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.867688:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.867692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.867695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.867699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.867708:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.867714:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.867734:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760738.867836:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23748608<>23752704 00000100:00100000:3.0:1789760738.867923:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.913782:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973117 gen 3 00000008:00010000:3.0:1789760738.913848:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760738.913856:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.913866:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.913887:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760738.913917:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.914012:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5798, 5798] 00000080:00200000:2.0:1789760738.914027:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.914033:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.914050:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23752704 i_blocks 47111 00000020:00200000:2.0:1789760738.914053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.914064:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.914109:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23752704, count: 4096 00000080:00200000:2.0:1789760738.914118:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23752704, bytes: 4096 00000080:00200000:2.0:1789760738.914128:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.914136:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.914138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23752704, 23756800) 0 0 00000080:00200000:2.0:1789760738.914143:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.914159:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5799, 5799] 00020000:00200000:2.0:1789760738.914169:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23752704 chunk: [23752704, 23756800) 23756800, 4096 00020000:00200000:2.0:1789760738.914173:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.914193:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5799, 5799] 00000008:00010000:2.0:1789760738.914212:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.914219:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.914230:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23752704, 23756800) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.914246:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23752704=0x16a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.914304:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.914307:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23756800 00000020:00010000:2.0:1789760738.914315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.914319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.914322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.914326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.914329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.914337:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.914345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.914372:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760738.914641:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23752704<>23756800 00000100:00100000:3.0:1789760738.914750:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724544:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760738.957637:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973118 gen 3 00000008:00010000:3.0:1789760738.957695:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760738.957703:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760738.957715:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760738.957734:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760738.957762:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724544:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760738.958023:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5799, 5799] 00000080:00200000:2.0:1789760738.958040:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.958046:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.958060:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23756800 i_blocks 47111 00000020:00200000:2.0:1789760738.958064:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.958074:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.958108:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23756800, count: 4096 00000080:00200000:2.0:1789760738.958115:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23756800, bytes: 4096 00000080:00200000:2.0:1789760738.958125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.958134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.958137:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23756800, 23760896) 0 0 00000080:00200000:2.0:1789760738.958141:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.958157:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5800, 5800] 00020000:00200000:2.0:1789760738.958165:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23756800 chunk: [23756800, 23760896) 23760896, 4096 00020000:00200000:2.0:1789760738.958169:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.958188:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5800, 5800] 00000008:00010000:2.0:1789760738.958205:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.958212:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760738.958225:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23756800, 23760896) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.958238:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23756800=0x16a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.958295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.958298:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23760896 00000020:00010000:2.0:1789760738.958307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.958310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.958313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.958317:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.958320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.958328:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.958335:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.958364:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760738.958505:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23756800<>23760896 00000100:00100000:3.0:1789760738.958597:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760738.958615:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.019049:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973119 gen 3 00000008:00010000:3.0:1789760739.019099:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760739.019108:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.019120:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.019185:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760739.019215:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724672:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.019299:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5800, 5800] 00000080:00200000:2.0:1789760739.019315:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.019321:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.019336:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23760896 i_blocks 47111 00000020:00200000:2.0:1789760739.019340:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.019478:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.019512:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23760896, count: 4096 00000080:00200000:2.0:1789760739.019520:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23760896, bytes: 4096 00000080:00200000:2.0:1789760739.019529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.019537:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.019539:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23760896, 23764992) 0 0 00000080:00200000:2.0:1789760739.019543:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.019558:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5801, 5801] 00020000:00200000:2.0:1789760739.019566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23760896 chunk: [23760896, 23764992) 23764992, 4096 00020000:00200000:2.0:1789760739.019570:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.019590:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5801, 5801] 00000008:00010000:2.0:1789760739.019608:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.019614:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.019625:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23760896, 23764992) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.019640:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23760896=0x16a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.019696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.019699:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23764992 00000020:00010000:2.0:1789760739.019707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.019710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.019713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.019717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.019720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.019728:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.019734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.019749:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760739.020074:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23760896<>23764992 00000100:00100000:3.0:1789760739.020139:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724800:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760739.063066:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760739.063081:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.063095:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.063118:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760739.063148:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493724800:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.063190:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5801, 5801] 00000080:00200000:2.0:1789760739.063207:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.063213:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.063238:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23764992 i_blocks 47111 00000020:00200000:2.0:1789760739.063242:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.063257:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.063290:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23764992, count: 4096 00000080:00200000:2.0:1789760739.063297:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23764992, bytes: 4096 00000080:00200000:2.0:1789760739.063308:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.063317:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.063319:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23764992, 23769088) 0 0 00000080:00200000:2.0:1789760739.063324:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.063339:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5802, 5802] 00020000:00200000:2.0:1789760739.063346:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23764992 chunk: [23764992, 23769088) 23769088, 4096 00020000:00200000:2.0:1789760739.063360:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.063378:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5802, 5802] 00000008:00010000:2.0:1789760739.063411:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.063417:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.063428:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23764992, 23769088) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.063440:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23764992=0x16aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.063490:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.063492:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23769088 00000020:00010000:2.0:1789760739.063501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.063504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.063508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.063511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.063514:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.063521:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.063527:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.063542:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760739.063632:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23764992<>23769088 00000100:00100000:3.0:1789760739.063797:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724928:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.098093:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973121 gen 3 00000008:00010000:3.0:1789760739.098154:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760739.098163:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.098173:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.098193:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760739.098223:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493724928:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.098536:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5802, 5802] 00000080:00200000:2.0:1789760739.098553:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.098559:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.098574:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23769088 i_blocks 47111 00000020:00200000:2.0:1789760739.098579:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.098590:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.098623:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23769088, count: 4096 00000080:00200000:2.0:1789760739.098631:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23769088, bytes: 4096 00000080:00200000:2.0:1789760739.098643:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.098652:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.098659:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23769088, 23773184) 0 0 00000080:00200000:2.0:1789760739.098663:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.098680:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5803, 5803] 00020000:00200000:2.0:1789760739.098688:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23769088 chunk: [23769088, 23773184) 23773184, 4096 00020000:00200000:2.0:1789760739.098692:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.098711:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5803, 5803] 00000008:00010000:2.0:1789760739.098728:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.098734:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.098745:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23769088, 23773184) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.098760:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23769088=0x16ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.099022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.099025:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23773184 00000020:00010000:2.0:1789760739.099035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.099038:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.099041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.099044:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.099047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.099056:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.099062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.099080:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760739.099180:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23769088<>23773184 00000100:00100000:3.0:1789760739.099251:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493725056:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760739.161726:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760739.161744:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.161757:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.161780:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760739.161811:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493725056:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760739.202913:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5804, 5804] 00000080:00200000:1.0:1789760739.202929:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.202936:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.202951:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23777280 i_blocks 47111 00000020:00200000:1.0:1789760739.202954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.202964:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.203002:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23777280, count: 4096 00000080:00200000:1.0:1789760739.203010:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23777280, bytes: 4096 00000080:00200000:1.0:1789760739.203019:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.203028:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.203030:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23777280, 23781376) 0 0 00000080:00200000:1.0:1789760739.203035:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.203050:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5805, 5805] 00020000:00200000:1.0:1789760739.203057:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23777280 chunk: [23777280, 23781376) 23781376, 4096 00020000:00200000:1.0:1789760739.203062:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.203081:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5805, 5805] 00000008:00010000:1.0:1789760739.203096:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.203102:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760739.203113:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23777280, 23781376) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.203126:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23777280=0x16ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.203178:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.203180:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23781376 00000020:00010000:1.0:1789760739.203187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.203190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.203193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.203196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.203199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.203208:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.203215:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.203229:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760739.203345:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 23777280<>23781376 00000100:00100000:1.0:1789760739.203434:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760739.203452:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493725312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760739.247729:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973124 gen 3 00000008:00010000:1.0:1789760739.247785:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92c00 00000020:00200000:1.0:1789760739.247795:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.247806:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.247874:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92c00 00000100:00100000:1.0:1789760739.247903:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493725312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760739.250454:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493725440:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760739.289061:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760739.289076:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.289086:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.289152:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760739.289182:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493725440:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760739.290125:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493725568:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760739.325236:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973125 gen 3 00000100:00100000:1.0:1789760739.325279:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ea00 x1876699493725568/t4294973126(4294973126) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760755 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760739.325327:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760739.325335:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.325344:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.325377:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760739.325660:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493725568:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760739.325742:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5807, 5807] 00000080:00200000:3.0:1789760739.325756:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.325760:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.325772:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23789568 i_blocks 47111 00000020:00200000:3.0:1789760739.325775:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.325783:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.325811:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23789568, count: 4096 00000080:00200000:3.0:1789760739.325817:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23789568, bytes: 4096 00000080:00200000:3.0:1789760739.328983:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.329004:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.329007:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23789568, 23793664) 0 0 00000080:00200000:3.0:1789760739.329015:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.329037:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5808, 5808] 00020000:00200000:3.0:1789760739.329048:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23789568 chunk: [23789568, 23793664) 23793664, 4096 00020000:00200000:3.0:1789760739.329053:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.329079:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5808, 5808] 00000008:00010000:3.0:1789760739.329103:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.329109:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760739.329123:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23789568, 23793664) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.329136:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23789568=0x16b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.329210:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.329213:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23793664 00000020:00010000:3.0:1789760739.329223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.329227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.329230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.329234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.329236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.329248:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.329255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.329282:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760739.331178:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23789568<>23793664 00000100:00100000:3.0:1789760739.331275:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.399028:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973127 gen 3 00000008:00010000:3.0:1789760739.399102:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760739.399112:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.399121:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.399151:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760739.399183:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.399328:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5808, 5808] 00000080:00200000:2.0:1789760739.399345:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.399360:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.399377:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23793664 i_blocks 47111 00000020:00200000:2.0:1789760739.399381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.399391:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.399671:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23793664, count: 4096 00000080:00200000:2.0:1789760739.399681:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23793664, bytes: 4096 00000080:00200000:2.0:1789760739.399696:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.399705:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.399707:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23793664, 23797760) 0 0 00000080:00200000:2.0:1789760739.399712:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.399729:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5809, 5809] 00020000:00200000:2.0:1789760739.399738:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23793664 chunk: [23793664, 23797760) 23797760, 4096 00020000:00200000:2.0:1789760739.399742:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.399762:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5809, 5809] 00000008:00010000:2.0:1789760739.399780:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.399787:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.399799:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23793664, 23797760) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.399812:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23793664=0x16b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.400062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.400065:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23797760 00000020:00010000:2.0:1789760739.400074:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.400077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.400081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.400085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.400088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.400096:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.400102:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.400120:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760739.400221:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23793664<>23797760 00000100:00100000:3.0:1789760739.400765:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760739.400788:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.449651:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973128 gen 3 00000008:00010000:3.0:1789760739.449701:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760739.449709:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.449720:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.449741:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760739.449769:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.450236:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5809, 5809] 00000080:00200000:2.0:1789760739.450253:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.450259:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.450273:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23797760 i_blocks 47111 00000020:00200000:2.0:1789760739.450277:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.450288:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.450320:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23797760, count: 4096 00000080:00200000:2.0:1789760739.450327:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23797760, bytes: 4096 00000080:00200000:2.0:1789760739.450336:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.450344:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.450357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23797760, 23801856) 0 0 00000080:00200000:2.0:1789760739.450361:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.450378:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5810, 5810] 00020000:00200000:2.0:1789760739.450387:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23797760 chunk: [23797760, 23801856) 23801856, 4096 00020000:00200000:2.0:1789760739.450392:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.450433:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5810, 5810] 00000008:00010000:2.0:1789760739.450461:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.450468:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.450491:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23797760, 23801856) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.450505:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23797760=0x16b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.450562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.450565:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23801856 00000020:00010000:2.0:1789760739.450573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.450577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.450580:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.450583:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.450586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.450595:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.450601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.450617:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760739.450718:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23797760<>23801856 00000100:00100000:3.0:1789760739.450891:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.509111:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973129 gen 3 00000008:00010000:3.0:1789760739.509165:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760739.509174:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.509187:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.509207:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760739.509237:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493725952:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.509355:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5810, 5810] 00000080:00200000:2.0:1789760739.509371:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.509377:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.509433:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23801856 i_blocks 47111 00000020:00200000:2.0:1789760739.509437:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.509449:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.509494:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23801856, count: 4096 00000080:00200000:2.0:1789760739.509502:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23801856, bytes: 4096 00000080:00200000:2.0:1789760739.509512:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.509521:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.509523:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23801856, 23805952) 0 0 00000080:00200000:2.0:1789760739.509529:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.509545:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5811, 5811] 00020000:00200000:2.0:1789760739.509553:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23801856 chunk: [23801856, 23805952) 23805952, 4096 00020000:00200000:2.0:1789760739.509558:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.509578:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5811, 5811] 00000008:00010000:2.0:1789760739.509596:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.509602:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.509614:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23801856, 23805952) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.509628:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23801856=0x16b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.509683:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.509686:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23805952 00000020:00010000:2.0:1789760739.509695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.509698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.509701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.509705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.509708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.509716:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.509722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.509738:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760739.509841:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23801856<>23805952 00000100:00100000:3.0:1789760739.510065:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760739.510084:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726080:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760739.565121:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760739.571135:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973130 gen 3 00000008:00010000:3.0:1789760739.571190:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760739.571199:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.571211:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.571232:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760739.571262:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.575215:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5811, 5811] 00000080:00200000:2.0:1789760739.575243:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.575249:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.575266:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23805952 i_blocks 47111 00000020:00200000:2.0:1789760739.575272:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.575291:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.575325:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23805952, count: 4096 00000080:00200000:2.0:1789760739.575333:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23805952, bytes: 4096 00000080:00200000:2.0:1789760739.575359:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.575368:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.575371:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23805952, 23810048) 0 0 00000080:00200000:2.0:1789760739.575375:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.575423:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5812, 5812] 00020000:00200000:2.0:1789760739.575432:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23805952 chunk: [23805952, 23810048) 23810048, 4096 00020000:00200000:2.0:1789760739.575437:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.575459:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5812, 5812] 00000008:00010000:2.0:1789760739.575479:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.575485:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.575499:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23805952, 23810048) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.575514:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23805952=0x16b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.575585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.575588:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23810048 00000020:00010000:2.0:1789760739.575596:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.575600:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.575603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.575606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.575609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.575619:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.575626:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.575647:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760739.575756:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23805952<>23810048 00000100:00100000:3.0:1789760739.576299:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.576329:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.626644:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973131 gen 3 00000008:00010000:3.0:1789760739.626694:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760739.626702:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.626713:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.626734:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760739.626764:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.627024:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5812, 5812] 00000080:00200000:2.0:1789760739.627041:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.627047:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.627064:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23810048 i_blocks 47111 00000020:00200000:2.0:1789760739.627068:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.627079:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.627113:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23810048, count: 4096 00000080:00200000:2.0:1789760739.627120:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23810048, bytes: 4096 00000080:00200000:2.0:1789760739.627130:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.627138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.627140:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23810048, 23814144) 0 0 00000080:00200000:2.0:1789760739.627144:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.627172:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5813, 5813] 00020000:00200000:2.0:1789760739.627181:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23810048 chunk: [23810048, 23814144) 23814144, 4096 00020000:00200000:2.0:1789760739.627185:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.627206:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5813, 5813] 00000008:00010000:2.0:1789760739.627223:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.627230:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.627241:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23810048, 23814144) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.627254:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23810048=0x16b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.627311:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.627314:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23814144 00000020:00010000:2.0:1789760739.627322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.627326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.627329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.627333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.627336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.627353:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.627361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.627377:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760739.627528:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23810048<>23814144 00000100:00100000:3.0:1789760739.627916:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760739.627939:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.666765:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973132 gen 3 00000008:00010000:3.0:1789760739.666814:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760739.667534:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.667547:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.667634:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760739.667666:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726336:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.670042:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5813, 5813] 00000080:00200000:2.0:1789760739.670071:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.670079:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.670098:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23814144 i_blocks 47111 00000020:00200000:2.0:1789760739.670103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.670120:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.670155:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23814144, count: 4096 00000080:00200000:2.0:1789760739.670164:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23814144, bytes: 4096 00000080:00200000:2.0:1789760739.670178:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.670188:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.670191:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23814144, 23818240) 0 0 00000080:00200000:2.0:1789760739.670195:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.670213:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5814, 5814] 00020000:00200000:2.0:1789760739.670222:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23814144 chunk: [23814144, 23818240) 23818240, 4096 00020000:00200000:2.0:1789760739.670227:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.670249:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5814, 5814] 00000008:00010000:2.0:1789760739.670268:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.670275:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.670287:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23814144, 23818240) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.670301:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23814144=0x16b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.671018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.671025:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23818240 00000020:00010000:2.0:1789760739.671035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.671039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.671042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.671046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.671048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.671058:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.671066:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.671093:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760739.671205:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23814144<>23818240 00000100:00100000:3.0:1789760739.671301:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726464:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.701134:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973133 gen 3 00000008:00010000:3.0:1789760739.701194:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760739.701202:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.701214:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.701264:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760739.701292:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726464:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.701470:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5814, 5814] 00000080:00200000:2.0:1789760739.701486:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.701492:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.701509:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23818240 i_blocks 47111 00000020:00200000:2.0:1789760739.701512:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.701522:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.701553:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23818240, count: 4096 00000080:00200000:2.0:1789760739.701559:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23818240, bytes: 4096 00000080:00200000:2.0:1789760739.701569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.701578:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.701580:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23818240, 23822336) 0 0 00000080:00200000:2.0:1789760739.701585:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.701599:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5815, 5815] 00020000:00200000:2.0:1789760739.701606:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23818240 chunk: [23818240, 23822336) 23822336, 4096 00020000:00200000:2.0:1789760739.701610:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.701627:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5815, 5815] 00000008:00010000:2.0:1789760739.701643:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.701648:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.701658:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23818240, 23822336) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.701671:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23818240=0x16b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.701723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.701725:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23822336 00000020:00010000:2.0:1789760739.701733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.701736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.701739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.701742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.701745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.701753:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.701758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.701771:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760739.702039:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23818240<>23822336 00000100:00100000:3.0:1789760739.702103:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.739024:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973134 gen 3 00000008:00010000:3.0:1789760739.739078:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760739.739087:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.739097:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.739243:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760739.739274:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726592:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760739.739975:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5815, 5815] 00000080:00200000:1.0:1789760739.739995:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.740001:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.740016:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23822336 i_blocks 47111 00000020:00200000:1.0:1789760739.740021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.740033:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.740062:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23822336, count: 4096 00000080:00200000:1.0:1789760739.740069:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23822336, bytes: 4096 00000080:00200000:1.0:1789760739.740080:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.740089:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.740091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23822336, 23826432) 0 0 00000080:00200000:1.0:1789760739.740096:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.740112:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5816, 5816] 00020000:00200000:1.0:1789760739.740122:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23822336 chunk: [23822336, 23826432) 23826432, 4096 00020000:00200000:1.0:1789760739.740127:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.740146:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5816, 5816] 00000008:00010000:1.0:1789760739.740163:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.740170:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760739.740183:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23822336, 23826432) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.740196:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23822336=0x16b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.740255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.740258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23826432 00000020:00010000:1.0:1789760739.740267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.740270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.740273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.740276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.740279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.740290:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.740297:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.740315:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760739.740644:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 23822336<>23826432 00000080:00200000:3.0:1789760739.789254:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5816, 5816] 00000080:00200000:3.0:1789760739.789270:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.789276:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.789292:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23826432 i_blocks 47111 00000020:00200000:3.0:1789760739.789297:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.789307:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.789564:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23826432, count: 4096 00000080:00200000:3.0:1789760739.789574:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23826432, bytes: 4096 00000080:00200000:3.0:1789760739.789586:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.789595:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.789597:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23826432, 23830528) 0 0 00000080:00200000:3.0:1789760739.789612:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.789630:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5817, 5817] 00020000:00200000:3.0:1789760739.789638:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23826432 chunk: [23826432, 23830528) 23830528, 4096 00020000:00200000:3.0:1789760739.789643:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.789664:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5817, 5817] 00000008:00010000:3.0:1789760739.789682:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.789688:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760739.789701:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23826432, 23830528) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.789714:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23826432=0x16b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.789771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.789774:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23830528 00000020:00010000:3.0:1789760739.789782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.789786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.789789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.789793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.789796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.789802:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.789808:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.789823:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760739.789941:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23826432<>23830528 00000100:00100000:3.0:1789760739.789987:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.790004:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.849100:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973136 gen 3 00000008:00010000:3.0:1789760739.849151:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760739.849160:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.849170:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.849191:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760739.849220:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493726848:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.849320:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5817, 5817] 00000080:00200000:2.0:1789760739.849338:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.849352:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.849368:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23830528 i_blocks 47111 00000020:00200000:2.0:1789760739.849373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.849383:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.849447:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23830528, count: 4096 00000080:00200000:2.0:1789760739.849455:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23830528, bytes: 4096 00000080:00200000:2.0:1789760739.849465:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.849481:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.849484:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23830528, 23834624) 0 0 00000080:00200000:2.0:1789760739.849488:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.849503:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5818, 5818] 00020000:00200000:2.0:1789760739.849518:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23830528 chunk: [23830528, 23834624) 23834624, 4096 00020000:00200000:2.0:1789760739.849523:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.849547:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5818, 5818] 00000008:00010000:2.0:1789760739.849563:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.849569:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.849580:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23830528, 23834624) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.849594:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23830528=0x16ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.849644:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.849647:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23834624 00000020:00010000:2.0:1789760739.849655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.849659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.849662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.849665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.849667:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.849676:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.849682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.849698:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760739.849795:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23830528<>23834624 00000100:00100000:2.0:1789760739.849832:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.849847:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.891476:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973137 gen 3 00000008:00010000:3.0:1789760739.891528:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760739.891537:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.891549:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.892884:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760739.892926:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493726976:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.893073:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5818, 5818] 00000080:00200000:2.0:1789760739.893090:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.893096:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.893114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23834624 i_blocks 47111 00000020:00200000:2.0:1789760739.893119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.893133:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.893169:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23834624, count: 4096 00000080:00200000:2.0:1789760739.893177:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23834624, bytes: 4096 00000080:00200000:2.0:1789760739.893189:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.893199:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.893201:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23834624, 23838720) 0 0 00000080:00200000:2.0:1789760739.893205:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.893223:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5819, 5819] 00020000:00200000:2.0:1789760739.893232:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23834624 chunk: [23834624, 23838720) 23838720, 4096 00020000:00200000:2.0:1789760739.893236:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.893256:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5819, 5819] 00000008:00010000:2.0:1789760739.893275:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.893282:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.893418:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23834624, 23838720) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.893435:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23834624=0x16bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.893504:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.893507:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23838720 00000020:00010000:2.0:1789760739.893517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.893521:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.893524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.893529:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.893532:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.893540:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.893547:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.893742:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760739.893852:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23834624<>23838720 00000100:00100000:3.0:1789760739.894030:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.894055:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727104:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760739.940499:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973138 gen 3 00000008:00010000:3.0:1789760739.940537:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760739.940543:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760739.940553:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760739.940573:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760739.940597:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727104:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760739.940679:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5819, 5819] 00000080:00200000:2.0:1789760739.940691:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.940695:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.940704:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23838720 i_blocks 47111 00000020:00200000:2.0:1789760739.940707:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.940715:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.940741:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23838720, count: 4096 00000080:00200000:2.0:1789760739.940747:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23838720, bytes: 4096 00000080:00200000:2.0:1789760739.940754:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.940761:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.940763:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23838720, 23842816) 0 0 00000080:00200000:2.0:1789760739.940767:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.940779:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5820, 5820] 00020000:00200000:2.0:1789760739.940785:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23838720 chunk: [23838720, 23842816) 23842816, 4096 00020000:00200000:2.0:1789760739.940788:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.940815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5820, 5820] 00000008:00010000:2.0:1789760739.940829:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.940834:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760739.940842:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23838720, 23842816) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.940853:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23838720=0x16bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.940894:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.940897:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23842816 00000020:00010000:2.0:1789760739.940903:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.940906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.940908:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.940911:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.940914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.940919:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.940924:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.940938:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760739.941010:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23838720<>23842816 00000100:00100000:3.0:1789760739.941100:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.010127:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973139 gen 3 00000008:00010000:3.0:1789760740.010182:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760740.010193:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.010205:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.010269:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760740.010301:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727232:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.010488:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5820, 5820] 00000080:00200000:2.0:1789760740.010505:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.010511:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.010527:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23842816 i_blocks 47111 00000020:00200000:2.0:1789760740.010531:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.010543:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.010576:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23842816, count: 4096 00000080:00200000:2.0:1789760740.010584:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23842816, bytes: 4096 00000080:00200000:2.0:1789760740.010594:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.010603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.010606:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23842816, 23846912) 0 0 00000080:00200000:2.0:1789760740.010611:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.010629:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5821, 5821] 00020000:00200000:2.0:1789760740.010638:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23842816 chunk: [23842816, 23846912) 23846912, 4096 00020000:00200000:2.0:1789760740.010642:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.010661:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5821, 5821] 00000008:00010000:2.0:1789760740.010680:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.010686:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.010699:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23842816, 23846912) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.010716:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23842816=0x16bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.010774:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.010778:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23846912 00000020:00010000:2.0:1789760740.010786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.010790:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.010793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.010797:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.010801:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.010808:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.010815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.010897:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760740.010996:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23842816<>23846912 00000100:00100000:3.0:1789760740.011064:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493727360:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.050085:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973140 gen 3 00000008:00010000:3.0:1789760740.050137:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760740.050146:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.050158:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.050179:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760740.050208:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493727360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.050526:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5821, 5821] 00000080:00200000:2.0:1789760740.050543:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.050549:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.050564:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23846912 i_blocks 47111 00000020:00200000:2.0:1789760740.050567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.050578:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.050612:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23846912, count: 4096 00000080:00200000:2.0:1789760740.050619:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23846912, bytes: 4096 00000080:00200000:2.0:1789760740.050627:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.050636:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.050638:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23846912, 23851008) 0 0 00000080:00200000:2.0:1789760740.050643:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.050658:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5822, 5822] 00020000:00200000:2.0:1789760740.050666:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23846912 chunk: [23846912, 23851008) 23851008, 4096 00020000:00200000:2.0:1789760740.050671:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.050690:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5822, 5822] 00000008:00010000:2.0:1789760740.050707:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.050713:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.050724:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23846912, 23851008) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.050737:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23846912=0x16be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.050793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.050796:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23851008 00000020:00010000:2.0:1789760740.050805:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.050808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.050811:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.050815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.050818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.050835:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.050842:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.050859:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760740.050965:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23846912<>23851008 00000100:00100000:3.0:1789760740.051033:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.104281:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973141 gen 3 00000008:00010000:3.0:1789760740.104331:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760740.104339:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.104360:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.104380:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760740.104458:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727488:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.105070:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5822, 5822] 00000080:00200000:2.0:1789760740.105088:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.105094:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.105111:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23851008 i_blocks 47111 00000020:00200000:2.0:1789760740.105116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.105127:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.105161:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23851008, count: 4096 00000080:00200000:2.0:1789760740.105168:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23851008, bytes: 4096 00000080:00200000:2.0:1789760740.105177:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.105186:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.105188:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23851008, 23855104) 0 0 00000080:00200000:2.0:1789760740.105193:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.105209:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5823, 5823] 00020000:00200000:2.0:1789760740.105217:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23851008 chunk: [23851008, 23855104) 23855104, 4096 00020000:00200000:2.0:1789760740.105221:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.105239:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5823, 5823] 00000008:00010000:2.0:1789760740.105256:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.105262:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.105274:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23851008, 23855104) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.105287:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23851008=0x16bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.105353:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.105356:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23855104 00000020:00010000:2.0:1789760740.105364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.105367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.105370:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.105375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.105377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.105385:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.105420:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.105437:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760740.108130:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23851008<>23855104 00000100:00100000:3.0:1789760740.108232:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760740.109598:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.158799:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973142 gen 3 00000008:00010000:3.0:1789760740.159273:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760740.159282:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.159293:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.159435:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760740.159467:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727616:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.159896:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23855104 i_blocks 47111 00000020:00200000:2.0:1789760740.159902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.159914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.159949:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23855104, count: 4096 00000080:00200000:2.0:1789760740.159957:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23855104, bytes: 4096 00000080:00200000:2.0:1789760740.159968:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.159977:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.159979:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23855104, 23859200) 0 0 00000080:00200000:2.0:1789760740.159984:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.160002:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5824, 5824] 00020000:00200000:2.0:1789760740.160072:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23855104 chunk: [23855104, 23859200) 23859200, 4096 00020000:00200000:2.0:1789760740.160079:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.160100:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5824, 5824] 00000008:00010000:2.0:1789760740.160119:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.160125:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.160137:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23855104, 23859200) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.160150:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23855104=0x16c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.160207:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.160210:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23859200 00000020:00010000:2.0:1789760740.160218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.160222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.160225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.160228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.160231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.160239:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.160247:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.160262:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760740.160755:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 23855104<>23859200 00000100:00100000:3.0:1789760740.161248:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727744:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.202811:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973143 gen 3 00000008:00010000:3.0:1789760740.202917:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760740.202926:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.202940:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.202999:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760740.203030:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493727744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.203170:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5824, 5824] 00000080:00200000:2.0:1789760740.203186:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.203392:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.203430:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23859200 i_blocks 47111 00000020:00200000:2.0:1789760740.203433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.203446:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.203481:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23859200, count: 4096 00000080:00200000:2.0:1789760740.203489:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23859200, bytes: 4096 00000080:00200000:2.0:1789760740.203500:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.203510:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.203513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23859200, 23863296) 0 0 00000080:00200000:2.0:1789760740.203518:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.203535:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5825, 5825] 00020000:00200000:2.0:1789760740.203545:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23859200 chunk: [23859200, 23863296) 23863296, 4096 00020000:00200000:2.0:1789760740.203549:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.203567:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5825, 5825] 00000008:00010000:2.0:1789760740.203585:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.203592:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.203604:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23859200, 23863296) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.203619:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23859200=0x16c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.203682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.203685:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23863296 00000020:00010000:2.0:1789760740.203696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.203700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.203704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.203708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.203712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.203719:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.203728:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.203745:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760740.203857:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 23859200<>23863296 00000100:00100000:3.0:1789760740.204218:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493727872:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.254307:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973144 gen 3 00000008:00010000:3.0:1789760740.254381:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760740.254391:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.254433:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.254520:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760740.254551:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493727872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.254654:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5825, 5825] 00000080:00200000:2.0:1789760740.254670:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.254676:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.254691:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23863296 i_blocks 47111 00000020:00200000:2.0:1789760740.254695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.254706:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.254738:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23863296, count: 4096 00000080:00200000:2.0:1789760740.254746:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23863296, bytes: 4096 00000080:00200000:2.0:1789760740.254757:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.254767:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.254769:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23863296, 23867392) 0 0 00000080:00200000:2.0:1789760740.254774:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.254791:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5826, 5826] 00020000:00200000:2.0:1789760740.254799:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23863296 chunk: [23863296, 23867392) 23867392, 4096 00020000:00200000:2.0:1789760740.254804:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.254894:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5826, 5826] 00000008:00010000:2.0:1789760740.254914:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.254920:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.254932:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23863296, 23867392) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.254946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23863296=0x16c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.255078:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.255081:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23867392 00000020:00010000:2.0:1789760740.255090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.255095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.255098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.255102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.255106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.255114:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.255122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.255138:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760740.255244:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 23863296<>23867392 00000100:00100000:3.0:1789760740.255631:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760740.255649:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.301390:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973145 gen 3 00000008:00010000:3.0:1789760740.301466:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760740.301475:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.301487:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.301510:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760740.301538:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.302190:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5826, 5826] 00000080:00200000:2.0:1789760740.302208:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.302215:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.302232:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23867392 i_blocks 47111 00000020:00200000:2.0:1789760740.302236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.302248:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.302283:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23867392, count: 4096 00000080:00200000:2.0:1789760740.302290:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23867392, bytes: 4096 00000080:00200000:2.0:1789760740.302299:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.302309:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.302311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23867392, 23871488) 0 0 00000080:00200000:2.0:1789760740.302316:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.302334:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5827, 5827] 00020000:00200000:2.0:1789760740.302355:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23867392 chunk: [23867392, 23871488) 23871488, 4096 00020000:00200000:2.0:1789760740.302360:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.302383:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5827, 5827] 00000008:00010000:2.0:1789760740.302428:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.302436:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.302448:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23867392, 23871488) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.302462:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23867392=0x16c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.302518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.302521:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23871488 00000020:00010000:2.0:1789760740.302531:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.302534:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.302537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.302541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.302545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.302552:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.302559:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.302574:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760740.302677:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 23867392<>23871488 00000100:00100000:3.0:1789760740.303997:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.337192:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973146 gen 3 00000008:00010000:3.0:1789760740.337778:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760740.337786:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.337796:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.337814:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760740.337841:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728128:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760740.339717:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493728256:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760740.383190:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973147 gen 3 00000008:00010000:1.0:1789760740.383226:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760740.383233:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.383241:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.383260:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760740.383287:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493728256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.383389:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5828, 5828] 00000080:00200000:2.0:1789760740.383432:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.383437:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.383450:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23875584 i_blocks 47111 00000020:00200000:2.0:1789760740.383453:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.383461:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.383487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23875584, count: 4096 00000080:00200000:2.0:1789760740.383493:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23875584, bytes: 4096 00000080:00200000:2.0:1789760740.383501:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.383509:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.383511:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23875584, 23879680) 0 0 00000080:00200000:2.0:1789760740.383515:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.383529:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5829, 5829] 00020000:00200000:2.0:1789760740.383537:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23875584 chunk: [23875584, 23879680) 23879680, 4096 00020000:00200000:2.0:1789760740.383540:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.383557:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5829, 5829] 00000008:00010000:2.0:1789760740.383571:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.383576:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.383584:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23875584, 23879680) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.383595:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23875584=0x16c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.383708:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.383712:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23879680 00000020:00010000:2.0:1789760740.383879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.383884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.383886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.383889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.383892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.383899:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.383905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.383920:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760740.384018:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 23875584<>23879680 00000100:00100000:3.0:1789760740.384216:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760740.384231:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.450598:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973148 gen 3 00000008:00010000:3.0:1789760740.450647:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760740.450655:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.450667:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.450691:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760740.450720:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.451052:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5829, 5829] 00000080:00200000:2.0:1789760740.451071:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.451077:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.451092:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23879680 i_blocks 47111 00000020:00200000:2.0:1789760740.451095:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.451106:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.451140:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23879680, count: 4096 00000080:00200000:2.0:1789760740.451148:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23879680, bytes: 4096 00000080:00200000:2.0:1789760740.451158:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.451166:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.451169:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23879680, 23883776) 0 0 00000080:00200000:2.0:1789760740.451174:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.451191:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5830, 5830] 00020000:00200000:2.0:1789760740.451200:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23879680 chunk: [23879680, 23883776) 23883776, 4096 00020000:00200000:2.0:1789760740.451204:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.451225:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5830, 5830] 00000008:00010000:2.0:1789760740.451243:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.451249:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.451331:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23879680, 23883776) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.451355:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23879680=0x16c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.451570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.451574:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23883776 00000020:00010000:2.0:1789760740.451585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.451589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.451592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.451595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.451598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.451605:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.451613:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.451630:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760740.451737:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 23879680<>23883776 00000100:00100000:3.0:1789760740.451956:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760740.451976:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728512:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760740.488796:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760740.488814:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.488872:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.488893:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760740.488918:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.489150:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5830, 5830] 00000080:00200000:2.0:1789760740.489163:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.489168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.489182:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23883776 i_blocks 47111 00000020:00200000:2.0:1789760740.489185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.489194:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.489222:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23883776, count: 4096 00000080:00200000:2.0:1789760740.489228:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23883776, bytes: 4096 00000080:00200000:2.0:1789760740.489236:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.489243:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.489244:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23883776, 23887872) 0 0 00000080:00200000:2.0:1789760740.489248:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.489309:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5831, 5831] 00020000:00200000:2.0:1789760740.489316:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23883776 chunk: [23883776, 23887872) 23887872, 4096 00020000:00200000:2.0:1789760740.489319:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.489335:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5831, 5831] 00000008:00010000:2.0:1789760740.489365:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.489371:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.489383:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23883776, 23887872) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.489418:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23883776=0x16c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.489545:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.489548:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23887872 00000020:00010000:2.0:1789760740.489558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.489561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.489565:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.489569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.489572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.489580:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.489587:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.489603:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760740.489709:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 23883776<>23887872 00000100:00100000:3.0:1789760740.489812:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.530991:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973150 gen 3 00000008:00010000:3.0:1789760740.531056:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760740.531064:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.531077:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.531096:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760740.531126:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.531255:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5831, 5831] 00000080:00200000:2.0:1789760740.531272:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.531279:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.531296:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23887872 i_blocks 47111 00000020:00200000:2.0:1789760740.531299:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.531309:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.531350:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23887872, count: 4096 00000080:00200000:2.0:1789760740.531356:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23887872, bytes: 4096 00000080:00200000:2.0:1789760740.531365:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.531374:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.531376:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23887872, 23891968) 0 0 00000080:00200000:2.0:1789760740.531380:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.531432:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5832, 5832] 00020000:00200000:2.0:1789760740.531441:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23887872 chunk: [23887872, 23891968) 23891968, 4096 00020000:00200000:2.0:1789760740.531445:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.531465:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5832, 5832] 00000008:00010000:2.0:1789760740.531481:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.531489:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.531501:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23887872, 23891968) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.531515:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23887872=0x16c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.531581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.531584:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23891968 00000020:00010000:2.0:1789760740.531592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.531596:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.531599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.531603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.531605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.531613:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.531619:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.531634:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760740.531733:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 23887872<>23891968 00000100:00100000:3.0:1789760740.532107:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728768:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760740.565640:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760740.565661:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.565674:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.565692:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760740.565729:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493728768:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.565904:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5832, 5832] 00000080:00200000:2.0:1789760740.565920:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.565926:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.565941:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23891968 i_blocks 47111 00000020:00200000:2.0:1789760740.565946:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.565957:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.565989:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23891968, count: 4096 00000080:00200000:2.0:1789760740.565996:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23891968, bytes: 4096 00000080:00200000:2.0:1789760740.566005:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.566012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.566013:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23891968, 23896064) 0 0 00000080:00200000:2.0:1789760740.566017:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.566030:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5833, 5833] 00020000:00200000:2.0:1789760740.566037:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23891968 chunk: [23891968, 23896064) 23896064, 4096 00020000:00200000:2.0:1789760740.566040:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.566056:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5833, 5833] 00000008:00010000:2.0:1789760740.566071:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.566075:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.566085:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23891968, 23896064) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.566098:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23891968=0x16c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.566152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.566155:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23896064 00000020:00010000:2.0:1789760740.566163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.566167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.566170:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.566174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.566177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.566185:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.566192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.566208:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760740.566298:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 23891968<>23896064 00000100:00100000:3.0:1789760740.566356:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728896:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.602456:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973152 gen 3 00000008:00010000:3.0:1789760740.602521:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760740.602530:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.602541:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.602563:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760740.602594:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493728896:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.602730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5833, 5833] 00000080:00200000:2.0:1789760740.602747:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.602753:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.602770:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23896064 i_blocks 47111 00000020:00200000:2.0:1789760740.602773:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.602784:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.602818:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23896064, count: 4096 00000080:00200000:2.0:1789760740.602825:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23896064, bytes: 4096 00000080:00200000:2.0:1789760740.602834:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.602844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.602846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23896064, 23900160) 0 0 00000080:00200000:2.0:1789760740.602851:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.602866:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5834, 5834] 00020000:00200000:2.0:1789760740.602874:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23896064 chunk: [23896064, 23900160) 23900160, 4096 00020000:00200000:2.0:1789760740.602878:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.602899:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5834, 5834] 00000008:00010000:2.0:1789760740.602917:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.602924:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.602937:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23896064, 23900160) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.602950:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23896064=0x16ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.603147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.603149:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23900160 00000020:00010000:2.0:1789760740.603158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.603162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.603165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.603168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.603171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.603179:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.603187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.603204:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760740.603304:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 23896064<>23900160 00000100:00100000:3.0:1789760740.603426:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729024:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760740.644332:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760740.644360:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.644372:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.644391:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000080:00200000:2.0:1789760740.644421:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5834, 5834] 00000080:00200000:2.0:1789760740.644439:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.644446:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.644460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23900160 i_blocks 47111 00000020:00200000:2.0:1789760740.644464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.644474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.644507:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23900160, count: 4096 00000100:00100000:3.0:1789760740.644508:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.644516:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23900160, bytes: 4096 00000080:00200000:2.0:1789760740.644525:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.644533:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.644535:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23900160, 23904256) 0 0 00000080:00200000:2.0:1789760740.644540:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.644557:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5835, 5835] 00020000:00200000:2.0:1789760740.644566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23900160 chunk: [23900160, 23904256) 23904256, 4096 00020000:00200000:2.0:1789760740.644570:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.644590:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5835, 5835] 00000008:00010000:2.0:1789760740.644609:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.644615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.644628:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23900160, 23904256) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.644642:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23900160=0x16cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.644795:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.644799:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23904256 00000020:00010000:2.0:1789760740.644808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.644812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.644815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.644947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.644950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.644959:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.644969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.644984:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760740.645085:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 23900160<>23904256 00000100:00100000:3.0:1789760740.645430:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760740.645449:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.676773:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973154 gen 3 00000008:00010000:3.0:1789760740.676894:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760740.676902:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.676914:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.676975:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760740.677005:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.677233:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5835, 5835] 00000080:00200000:2.0:1789760740.677250:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.677256:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.677272:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23904256 i_blocks 47111 00000020:00200000:2.0:1789760740.677276:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.677287:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.677323:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23904256, count: 4096 00000080:00200000:2.0:1789760740.677330:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23904256, bytes: 4096 00000080:00200000:2.0:1789760740.677349:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.677359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.677361:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23904256, 23908352) 0 0 00000080:00200000:2.0:1789760740.677366:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.677382:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5836, 5836] 00020000:00200000:2.0:1789760740.677486:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23904256 chunk: [23904256, 23908352) 23908352, 4096 00020000:00200000:2.0:1789760740.677492:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.677512:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5836, 5836] 00000008:00010000:2.0:1789760740.677532:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.677538:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.677552:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23904256, 23908352) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.677565:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23904256=0x16cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.677623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.677626:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23908352 00000020:00010000:2.0:1789760740.677635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.677638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.677642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.677646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.677649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.677657:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.677664:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.677681:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760740.677785:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 23904256<>23908352 00000100:00100000:3.0:1789760740.677955:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729280:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.719742:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973155 gen 3 00000008:00010000:3.0:1789760740.719792:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760740.719801:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.719815:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.719835:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760740.719861:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.720111:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5836, 5836] 00000080:00200000:2.0:1789760740.720127:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.720133:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.720148:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23908352 i_blocks 47111 00000020:00200000:2.0:1789760740.720151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.720162:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.720193:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23908352, count: 4096 00000080:00200000:2.0:1789760740.720200:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23908352, bytes: 4096 00000080:00200000:2.0:1789760740.720211:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.720219:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.720222:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23908352, 23912448) 0 0 00000080:00200000:2.0:1789760740.720226:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.720240:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5837, 5837] 00020000:00200000:2.0:1789760740.720248:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23908352 chunk: [23908352, 23912448) 23912448, 4096 00020000:00200000:2.0:1789760740.720253:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.720271:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5837, 5837] 00000008:00010000:2.0:1789760740.720287:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.720294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.720305:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23908352, 23912448) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.720327:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23908352=0x16cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.720668:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.720671:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23912448 00000020:00010000:2.0:1789760740.720691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.720695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.720698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.720702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.720705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.720712:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.720720:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.720746:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760740.720847:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23908352<>23912448 00000100:00100000:3.0:1789760740.721097:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.769760:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973156 gen 3 00000008:00010000:3.0:1789760740.769813:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760740.769832:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.769845:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.769862:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760740.769901:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.770327:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5837, 5837] 00000080:00200000:2.0:1789760740.770357:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.770364:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.770381:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23912448 i_blocks 47111 00000020:00200000:2.0:1789760740.770385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.770439:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.770474:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23912448, count: 4096 00000080:00200000:2.0:1789760740.770482:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23912448, bytes: 4096 00000080:00200000:2.0:1789760740.770493:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.770502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.770505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23912448, 23916544) 0 0 00000080:00200000:2.0:1789760740.770510:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.770528:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5838, 5838] 00020000:00200000:2.0:1789760740.770537:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23912448 chunk: [23912448, 23916544) 23916544, 4096 00020000:00200000:2.0:1789760740.770542:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.770562:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5838, 5838] 00000008:00010000:2.0:1789760740.770580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.770592:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.770606:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23912448, 23916544) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.770622:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23912448=0x16ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.770675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.770677:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23916544 00000020:00010000:2.0:1789760740.770686:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.770690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.770693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.770696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.770699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.770707:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.770714:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.770810:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:3.0:1789760740.770937:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23912448<>23916544 00000100:00100000:3.0:1789760740.770999:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729536:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.821473:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973157 gen 3 00000008:00010000:3.0:1789760740.821526:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760740.821535:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.821549:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.821570:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760740.821599:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.821639:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5838, 5838] 00000080:00200000:2.0:1789760740.821657:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.821663:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.821679:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23916544 i_blocks 47111 00000020:00200000:2.0:1789760740.821683:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.821694:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.821725:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23916544, count: 4096 00000080:00200000:2.0:1789760740.821732:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23916544, bytes: 4096 00000080:00200000:2.0:1789760740.821741:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.821749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.821751:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23916544, 23920640) 0 0 00000080:00200000:2.0:1789760740.821755:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.821771:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5839, 5839] 00020000:00200000:2.0:1789760740.821779:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23916544 chunk: [23916544, 23920640) 23920640, 4096 00020000:00200000:2.0:1789760740.821783:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.821801:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5839, 5839] 00000008:00010000:2.0:1789760740.821819:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.821824:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.821836:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23916544, 23920640) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.821849:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23916544=0x16cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.821899:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.821901:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23920640 00000020:00010000:2.0:1789760740.821910:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.821913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.821916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.821920:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.821923:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.821931:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.821937:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.821953:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760740.822269:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23916544<>23920640 00000100:00100000:3.0:1789760740.822707:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.871052:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973158 gen 3 00000008:00010000:3.0:1789760740.871105:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760740.871113:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.871126:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.871146:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760740.871178:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729664:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.871813:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5839, 5839] 00000080:00200000:2.0:1789760740.871831:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.871837:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.871852:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23920640 i_blocks 47111 00000020:00200000:2.0:1789760740.871857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.871869:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.871908:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23920640, count: 4096 00000080:00200000:2.0:1789760740.871916:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23920640, bytes: 4096 00000080:00200000:2.0:1789760740.871928:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.871938:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.871941:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23920640, 23924736) 0 0 00000080:00200000:2.0:1789760740.871945:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.871963:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5840, 5840] 00020000:00200000:2.0:1789760740.871972:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23920640 chunk: [23920640, 23924736) 23924736, 4096 00020000:00200000:2.0:1789760740.871976:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.871996:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5840, 5840] 00000008:00010000:2.0:1789760740.872015:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.872022:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.872034:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23920640, 23924736) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.872050:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23920640=0x16d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.872110:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.872114:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23924736 00000020:00010000:2.0:1789760740.872121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.872125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.872129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.872133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.872136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.872144:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.872151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.872167:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760740.872267:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23920640<>23924736 00000100:00100000:3.0:1789760740.872494:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760740.872516:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729792:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.921121:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973159 gen 3 00000008:00010000:3.0:1789760740.921159:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760740.921166:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.921175:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.921239:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760740.921266:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493729792:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.921349:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5840, 5840] 00000080:00200000:2.0:1789760740.921362:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.921367:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.921527:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23924736 i_blocks 47111 00000020:00200000:2.0:1789760740.921531:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.921541:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.921568:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23924736, count: 4096 00000080:00200000:2.0:1789760740.921574:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23924736, bytes: 4096 00000080:00200000:2.0:1789760740.921582:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.921590:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.921592:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23924736, 23928832) 0 0 00000080:00200000:2.0:1789760740.921596:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.921608:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5841, 5841] 00020000:00200000:2.0:1789760740.921615:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23924736 chunk: [23924736, 23928832) 23928832, 4096 00020000:00200000:2.0:1789760740.921620:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.921636:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5841, 5841] 00000008:00010000:2.0:1789760740.921648:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.921653:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.921662:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23924736, 23928832) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.921671:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23924736=0x16d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.921727:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.921730:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23928832 00000020:00010000:2.0:1789760740.921737:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.921740:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.921743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.921746:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.921748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.921754:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.921759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.921773:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760740.921878:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23924736<>23928832 00000100:00100000:2.0:1789760740.921915:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760740.921929:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760740.966967:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973160 gen 3 00000008:00010000:3.0:1789760740.967001:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760740.967007:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760740.967015:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760740.967202:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760740.967229:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493729920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760740.967550:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5841, 5841] 00000080:00200000:2.0:1789760740.967572:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.967577:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.967589:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23928832 i_blocks 47111 00000020:00200000:2.0:1789760740.967592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.967602:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.967635:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23928832, count: 4096 00000080:00200000:2.0:1789760740.967642:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23928832, bytes: 4096 00000080:00200000:2.0:1789760740.967652:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.967659:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.967662:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23928832, 23932928) 0 0 00000080:00200000:2.0:1789760740.967667:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.967682:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5842, 5842] 00020000:00200000:2.0:1789760740.967690:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23928832 chunk: [23928832, 23932928) 23932928, 4096 00020000:00200000:2.0:1789760740.967694:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.967771:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5842, 5842] 00000008:00010000:2.0:1789760740.967786:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.967792:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760740.967801:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23928832, 23932928) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.967812:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23928832=0x16d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.967859:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.967862:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23932928 00000020:00010000:2.0:1789760740.967869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.967872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.967907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.967911:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.967914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.967922:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.967927:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.967941:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760740.968014:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 23928832<>23932928 00000100:00100000:2.0:1789760740.968058:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760740.968074:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493730048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760741.018152:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973161 gen 3 00000008:00010000:3.0:1789760741.018191:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760741.018198:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.018207:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.018283:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760741.018320:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493730048:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760741.067197:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5843, 5843] 00000080:00200000:2.0:1789760741.067211:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.067327:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.067347:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23937024 i_blocks 47111 00000020:00200000:2.0:1789760741.067351:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.067516:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.067544:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23937024, count: 4096 00000080:00200000:2.0:1789760741.067551:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23937024, bytes: 4096 00000080:00200000:2.0:1789760741.067562:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.067569:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.067571:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23937024, 23941120) 0 0 00000080:00200000:2.0:1789760741.067576:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.067590:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5844, 5844] 00020000:00200000:2.0:1789760741.067596:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23937024 chunk: [23937024, 23941120) 23941120, 4096 00020000:00200000:2.0:1789760741.067600:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.067616:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5844, 5844] 00000008:00010000:2.0:1789760741.067630:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.067635:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.067643:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23937024, 23941120) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.067655:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23937024=0x16d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.067706:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.067708:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23941120 00000020:00010000:2.0:1789760741.067715:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.067718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.067720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.067724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.067726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.067733:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.067738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.067758:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760741.067883:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 23937024<>23941120 00000100:00100000:3.0:1789760741.067959:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493730304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760741.107641:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973163 gen 3 00000008:00010000:3.0:1789760741.107687:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760741.107693:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.107702:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.107719:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760741.107745:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493730304:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760741.183755:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5845, 5845] 00000080:00200000:3.0:1789760741.183767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.183772:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.183783:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23945216 i_blocks 47111 00000020:00200000:3.0:1789760741.183787:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.183795:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760741.183821:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23945216, count: 4096 00000080:00200000:3.0:1789760741.183828:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23945216, bytes: 4096 00000080:00200000:3.0:1789760741.183837:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760741.183844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760741.183846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23945216, 23949312) 0 0 00000080:00200000:3.0:1789760741.183850:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760741.183862:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5846, 5846] 00020000:00200000:3.0:1789760741.183869:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23945216 chunk: [23945216, 23949312) 23949312, 4096 00020000:00200000:3.0:1789760741.183872:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760741.183887:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5846, 5846] 00000008:00010000:3.0:1789760741.183900:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760741.183905:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760741.183914:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23945216, 23949312) append 0 append lockpos 0 00000080:00200000:3.0:1789760741.183924:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23945216=0x16d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760741.183965:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760741.183968:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23949312 00000020:00010000:3.0:1789760741.183974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760741.183977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.183980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.183982:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760741.183985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760741.183991:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760741.184061:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760741.184075:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760741.184148:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 23945216<>23949312 00000100:00100000:2.0:1789760741.184207:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493730560:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760741.272157:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760741.272168:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.272176:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.272197:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760741.272222:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493730560:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.272711:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760741.272727:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493730688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.321281:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973166 gen 3 00000008:00010000:1.0:1789760741.321342:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760741.321349:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.321359:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.321386:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760741.321447:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493730688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.322493:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493730816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.386549:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973167 gen 3 00000008:00010000:1.0:1789760741.386603:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760741.386612:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.386624:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.386728:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760741.387533:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493730816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.389192:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493730944:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.431669:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973168 gen 3 00000008:00010000:1.0:1789760741.431721:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760741.431729:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.431739:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.432005:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760741.432036:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493730944:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760741.432503:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5849, 5849] 00000080:00200000:3.0:1789760741.432521:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.432528:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.432544:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23961600 i_blocks 47111 00000020:00200000:3.0:1789760741.432548:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.432558:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760741.432588:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23961600, count: 4096 00000080:00200000:3.0:1789760741.432595:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23961600, bytes: 4096 00000080:00200000:3.0:1789760741.432605:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760741.432613:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760741.432615:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23961600, 23965696) 0 0 00000080:00200000:3.0:1789760741.432620:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760741.432695:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5850, 5850] 00020000:00200000:3.0:1789760741.432705:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23961600 chunk: [23961600, 23965696) 23965696, 4096 00020000:00200000:3.0:1789760741.432709:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760741.432728:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5850, 5850] 00000008:00010000:3.0:1789760741.432745:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760741.432750:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760741.432761:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23961600, 23965696) append 0 append lockpos 0 00000080:00200000:3.0:1789760741.432773:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23961600=0x16da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760741.433320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760741.433324:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23965696 00000020:00010000:3.0:1789760741.433345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760741.433349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.433352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.433356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760741.433358:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760741.433367:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760741.433374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760741.433431:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760741.433538:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 23961600<>23965696 00000100:00100000:2.0:1789760741.433626:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493731072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760741.473593:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973169 gen 3 00000008:00010000:3.0:1789760741.473651:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760741.473659:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.473667:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.473686:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760741.473711:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493731072:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.475115:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760741.475137:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731200:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760741.517106:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760741.517122:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.517134:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.517160:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760741.517190:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731200:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.518080:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731328:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.557611:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973171 gen 3 00000008:00010000:1.0:1789760741.557690:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760741.557698:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.557709:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.557731:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760741.557762:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760741.557915:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5852, 5852] 00000080:00200000:2.0:1789760741.557932:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.557938:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.557954:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23973888 i_blocks 47111 00000020:00200000:2.0:1789760741.557957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.557968:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.557999:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23973888, count: 4096 00000080:00200000:2.0:1789760741.558006:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23973888, bytes: 4096 00000080:00200000:2.0:1789760741.558014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.558023:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.558026:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23973888, 23977984) 0 0 00000080:00200000:2.0:1789760741.558030:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.558046:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5853, 5853] 00020000:00200000:2.0:1789760741.558055:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23973888 chunk: [23973888, 23977984) 23977984, 4096 00020000:00200000:2.0:1789760741.558060:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.558094:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5853, 5853] 00000008:00010000:2.0:1789760741.558111:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.558118:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.558131:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23973888, 23977984) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.558144:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23973888=0x16dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.558197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.558201:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23977984 00000020:00010000:2.0:1789760741.558209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.558213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.558216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.558219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.558222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.558229:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.558236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.558251:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760741.558374:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 23973888<>23977984 00000100:00100000:3.0:1789760741.558613:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493731456:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760741.596752:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760741.596768:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.596782:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.596867:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760741.596899:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493731456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760741.599349:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5853, 5853] 00000080:00200000:2.0:1789760741.600256:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.600266:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760741.601865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493731584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.649923:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973173 gen 3 00000008:00010000:1.0:1789760741.650115:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760741.650125:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.650136:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.650220:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760741.650251:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493731584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.651235:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731712:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.687113:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973174 gen 3 00000008:00010000:1.0:1789760741.687164:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760741.687172:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760741.687196:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760741.687293:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760741.687323:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493731712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760741.687646:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5855, 5855] 00000080:00200000:2.0:1789760741.687661:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.687666:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760741.687744:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23986176 i_blocks 47111 00000020:00200000:1.0:1789760741.687747:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760741.687756:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760741.687790:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23986176, count: 4096 00000080:00200000:1.0:1789760741.687796:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23986176, bytes: 4096 00000080:00200000:1.0:1789760741.687810:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760741.687820:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760741.687822:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23986176, 23990272) 0 0 00000080:00200000:1.0:1789760741.687827:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760741.687843:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5856, 5856] 00020000:00200000:1.0:1789760741.687851:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23986176 chunk: [23986176, 23990272) 23990272, 4096 00020000:00200000:1.0:1789760741.687856:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760741.687876:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5856, 5856] 00000008:00010000:1.0:1789760741.687897:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760741.687903:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760741.687914:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23986176, 23990272) append 0 append lockpos 0 00000080:00200000:1.0:1789760741.687930:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23986176=0x16e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760741.687986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760741.687989:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23990272 00000020:00010000:1.0:1789760741.687997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:1.0:1789760741.688000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.688004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760741.688007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760741.688010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760741.688022:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760741.688028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760741.688077:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760741.688319:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 23986176<>23990272 00000080:00200000:2.0:1789760741.740142:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5856, 5856] 00000080:00200000:2.0:1789760741.740160:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.740166:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.740182:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23990272 i_blocks 47111 00000020:00200000:2.0:1789760741.740186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.740196:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.740226:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23990272, count: 4096 00000080:00200000:2.0:1789760741.740233:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23990272, bytes: 4096 00000080:00200000:2.0:1789760741.740244:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.740253:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.740255:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23990272, 23994368) 0 0 00000080:00200000:2.0:1789760741.740259:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.740276:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5857, 5857] 00020000:00200000:2.0:1789760741.740283:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23990272 chunk: [23990272, 23994368) 23994368, 4096 00020000:00200000:2.0:1789760741.740288:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.740308:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5857, 5857] 00000008:00010000:2.0:1789760741.740324:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.740341:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.740353:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23990272, 23994368) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.740366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23990272=0x16e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.740541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.740544:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23994368 00000020:00010000:2.0:1789760741.740552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.740555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.740558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.740561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.740564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.740571:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.740578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.740588:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760741.740688:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23990272<>23994368 00000100:00100000:3.0:1789760741.741209:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493731968:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760741.795654:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973176 gen 3 00000008:00010000:2.0:1789760741.795705:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760741.795714:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.795727:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.795754:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760741.795784:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493731968:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760741.796240:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5857, 5857] 00000080:00200000:3.0:1789760741.796256:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.796262:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.796277:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23994368 i_blocks 47111 00000020:00200000:3.0:1789760741.796280:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.796291:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760741.796323:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23994368, count: 4096 00000080:00200000:3.0:1789760741.796340:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23994368, bytes: 4096 00000080:00200000:3.0:1789760741.796352:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760741.796360:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760741.796363:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23994368, 23998464) 0 0 00000080:00200000:3.0:1789760741.796367:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760741.796465:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5858, 5858] 00020000:00200000:3.0:1789760741.796475:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23994368 chunk: [23994368, 23998464) 23998464, 4096 00020000:00200000:3.0:1789760741.796479:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760741.796501:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5858, 5858] 00000008:00010000:3.0:1789760741.796519:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760741.796525:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760741.796537:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23994368, 23998464) append 0 append lockpos 0 00000080:00200000:3.0:1789760741.797036:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23994368=0x16e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760741.797106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760741.797109:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23998464 00000020:00010000:3.0:1789760741.797118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:3.0:1789760741.797122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.797125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760741.797128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760741.797132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760741.797141:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760741.797149:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760741.797170:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760741.797276:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 23994368<>23998464 00000100:00100000:2.0:1789760741.797512:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732096:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760741.838557:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973177 gen 3 00000008:00010000:2.0:1789760741.838609:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760741.838618:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.838629:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.838654:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760741.838682:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732096:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760741.838729:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5858, 5858] 00000080:00200000:3.0:1789760741.838745:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.838751:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.838767:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23998464 i_blocks 47111 00000020:00200000:3.0:1789760741.838771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.838780:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760741.838818:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23998464, count: 4096 00000080:00200000:3.0:1789760741.838826:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23998464, bytes: 4096 00000080:00200000:3.0:1789760741.838840:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760741.838849:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760741.838852:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23998464, 24002560) 0 0 00000080:00200000:3.0:1789760741.838856:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.838933:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5859, 5859] 00020000:00200000:2.0:1789760741.838944:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23998464 chunk: [23998464, 24002560) 24002560, 4096 00020000:00200000:2.0:1789760741.838949:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.838971:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5859, 5859] 00000008:00010000:2.0:1789760741.838993:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.838999:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.839012:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23998464, 24002560) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.839028:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=23998464=0x16e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.839347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.839351:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24002560 00000020:00010000:2.0:1789760741.839360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.839364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.839367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.839371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac7b8 (ffff8eb6450b4758/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.839373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4758/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.839382:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.839389:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.839553:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760741.839663:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 23998464<>24002560 00000100:00100000:2.0:1789760741.839714:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760741.839894:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732224:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760741.868065:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760741.868081:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.868093:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.868115:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760741.868147:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732224:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760741.868152:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5859, 5859] 00000080:00200000:3.0:1789760741.868170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760741.868176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760741.868191:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24002560 i_blocks 47111 00000020:00200000:3.0:1789760741.868195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760741.868207:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.868610:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24002560, count: 4096 00000080:00200000:2.0:1789760741.868621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24002560, bytes: 4096 00000080:00200000:2.0:1789760741.868631:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.868640:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.868643:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24002560, 24006656) 0 0 00000080:00200000:2.0:1789760741.868648:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.868665:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5860, 5860] 00020000:00200000:2.0:1789760741.868673:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24002560 chunk: [24002560, 24006656) 24006656, 4096 00020000:00200000:2.0:1789760741.868678:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.868697:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5860, 5860] 00000008:00010000:2.0:1789760741.868716:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.868722:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.868734:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24002560, 24006656) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.870096:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24002560=0x16e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.870178:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.870182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24006656 00000020:00010000:2.0:1789760741.870193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.870196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.870200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.870204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.870206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.870219:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.870227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.870248:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760741.870376:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24002560<>24006656 00000100:00100000:3.0:1789760741.870481:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732352:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760741.902030:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973179 gen 3 00000008:00010000:2.0:1789760741.902094:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760741.902102:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.902112:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.902138:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760741.902165:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732352:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760741.933492:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732736:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760741.933816:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760741.933825:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493732608:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760741.934041:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493732864:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760741.936830:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732736:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760741.938617:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493732608:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760741.938638:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493732864:192.168.203.160@tcp:400:kworker.0 00000080:00200000:2.0:1789760741.945659:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5861, 5861] 00000080:00200000:2.0:1789760741.945675:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.945680:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.945696:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24010752 i_blocks 47111 00000020:00200000:2.0:1789760741.945699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.945710:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.945741:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24010752, count: 4096 00000080:00200000:2.0:1789760741.945747:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24010752, bytes: 4096 00000080:00200000:2.0:1789760741.945766:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.945774:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.945777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24010752, 24014848) 0 0 00000080:00200000:2.0:1789760741.945781:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.945796:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5862, 5862] 00020000:00200000:2.0:1789760741.945805:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24010752 chunk: [24010752, 24014848) 24014848, 4096 00020000:00200000:2.0:1789760741.945818:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.945839:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5862, 5862] 00000008:00010000:2.0:1789760741.945856:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.945863:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760741.945879:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24010752, 24014848) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.945893:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24010752=0x16e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.945952:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.945955:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24014848 00000020:00010000:2.0:1789760741.945963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.945966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.945969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.945973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.945975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.945982:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.945989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.946007:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760741.946107:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24010752<>24014848 00000100:00100000:3.0:1789760741.946173:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732992:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760741.977364:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973181 gen 3 00000008:00010000:2.0:1789760741.977449:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760741.977458:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760741.977470:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760741.977547:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760741.977575:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493732992:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760741.982539:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760741.983284:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493733120:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.035207:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973182 gen 3 00000008:00010000:1.0:1789760742.035258:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760742.035266:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.035277:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.035306:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760742.035348:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493733120:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760742.035560:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5863, 5863] 00000080:00200000:2.0:1789760742.035580:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.035587:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.035604:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24018944 i_blocks 47111 00000020:00200000:2.0:1789760742.035608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.035619:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.035652:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24018944, count: 4096 00000080:00200000:2.0:1789760742.035659:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24018944, bytes: 4096 00000080:00200000:2.0:1789760742.035669:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.035679:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.035681:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24018944, 24023040) 0 0 00000080:00200000:2.0:1789760742.035686:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.035702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5864, 5864] 00020000:00200000:2.0:1789760742.035711:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24018944 chunk: [24018944, 24023040) 24023040, 4096 00020000:00200000:2.0:1789760742.035715:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.035734:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5864, 5864] 00000008:00010000:2.0:1789760742.035751:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.035998:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760742.036013:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24018944, 24023040) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.036031:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24018944=0x16e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.036095:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.036099:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24023040 00000020:00010000:2.0:1789760742.036109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.036113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.036116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.036120:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.036123:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.036132:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.036140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.036159:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760742.036266:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24018944<>24023040 00000100:00100000:3.0:1789760742.036379:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493733248:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760742.092450:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973183 gen 3 00000008:00010000:2.0:1789760742.092522:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760742.092531:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760742.092544:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760742.092585:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760742.092610:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493733248:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760742.093063:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5864, 5864] 00000080:00200000:1.0:1789760742.093083:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.093089:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.093109:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24023040 i_blocks 47111 00000020:00200000:1.0:1789760742.093113:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.093127:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.093172:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24023040, count: 4096 00000080:00200000:1.0:1789760742.093180:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24023040, bytes: 4096 00000080:00200000:1.0:1789760742.093195:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.093206:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.093208:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24023040, 24027136) 0 0 00000080:00200000:1.0:1789760742.093213:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.093232:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5865, 5865] 00020000:00200000:1.0:1789760742.093240:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24023040 chunk: [24023040, 24027136) 24027136, 4096 00020000:00200000:1.0:1789760742.093245:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.093264:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5865, 5865] 00000008:00010000:1.0:1789760742.093282:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.093288:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.093301:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24023040, 24027136) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.093316:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24023040=0x16e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.093431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.093434:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24027136 00000020:00010000:1.0:1789760742.093442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.093446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.093449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.093453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.093456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.093467:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.093475:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.093492:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760742.093615:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 24023040<>24027136 00000080:00200000:1.0:1789760742.135638:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5865, 5865] 00000080:00200000:1.0:1789760742.135656:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.135663:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.135679:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24027136 i_blocks 47111 00000020:00200000:1.0:1789760742.135683:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.135693:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.135728:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24027136, count: 4096 00000080:00200000:1.0:1789760742.135736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24027136, bytes: 4096 00000080:00200000:1.0:1789760742.135746:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.135755:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.135758:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24027136, 24031232) 0 0 00000080:00200000:1.0:1789760742.135763:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.135780:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5866, 5866] 00020000:00200000:1.0:1789760742.135789:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24027136 chunk: [24027136, 24031232) 24031232, 4096 00020000:00200000:1.0:1789760742.135793:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.135813:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5866, 5866] 00000008:00010000:1.0:1789760742.135888:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.135894:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.135907:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24027136, 24031232) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.135922:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24027136=0x16ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.135980:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.135983:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24031232 00000020:00010000:1.0:1789760742.135991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.135994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.135997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.136001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.136003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.136014:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.136021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.136037:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760742.136139:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 24027136<>24031232 00000080:00200000:1.0:1789760742.172187:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5866, 5866] 00000080:00200000:1.0:1789760742.172204:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.172210:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.172224:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24031232 i_blocks 47111 00000020:00200000:1.0:1789760742.172228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.172238:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.172271:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24031232, count: 4096 00000080:00200000:1.0:1789760742.172278:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24031232, bytes: 4096 00000080:00200000:1.0:1789760742.172289:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.172299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.172301:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24031232, 24035328) 0 0 00000080:00200000:1.0:1789760742.172306:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.172322:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5867, 5867] 00020000:00200000:1.0:1789760742.172342:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24031232 chunk: [24031232, 24035328) 24035328, 4096 00020000:00200000:1.0:1789760742.172347:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.172471:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5867, 5867] 00000008:00010000:1.0:1789760742.172492:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.172499:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.172513:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24031232, 24035328) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.172528:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24031232=0x16eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.172597:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.172600:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24035328 00000020:00010000:1.0:1789760742.172610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.172614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.172617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.172621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.172635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.172648:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.172657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.172674:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760742.172789:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 24031232<>24035328 00000080:00200000:1.0:1789760742.203296:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5867, 5867] 00000080:00200000:1.0:1789760742.203311:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.203318:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.203342:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24035328 i_blocks 47111 00000020:00200000:1.0:1789760742.203345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.203356:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.203496:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24035328, count: 4096 00000080:00200000:1.0:1789760742.203506:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24035328, bytes: 4096 00000080:00200000:1.0:1789760742.203519:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.203528:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.203531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24035328, 24039424) 0 0 00000080:00200000:1.0:1789760742.203535:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.203552:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5868, 5868] 00020000:00200000:1.0:1789760742.203561:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24035328 chunk: [24035328, 24039424) 24039424, 4096 00020000:00200000:1.0:1789760742.203566:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.203586:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5868, 5868] 00000008:00010000:1.0:1789760742.203604:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.203610:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.203622:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24035328, 24039424) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.203636:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24035328=0x16ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.203693:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.203696:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24039424 00000020:00010000:1.0:1789760742.203704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.203708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.203711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.203715:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.203718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.203729:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.203737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.203787:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760742.203887:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 24035328<>24039424 00000080:00200000:1.0:1789760742.242693:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5868, 5868] 00000080:00200000:1.0:1789760742.242713:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.242719:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.242737:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24039424 i_blocks 47111 00000020:00200000:1.0:1789760742.242741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.242752:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.242789:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24039424, count: 4096 00000080:00200000:1.0:1789760742.242797:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24039424, bytes: 4096 00000080:00200000:1.0:1789760742.242808:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.242819:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.242821:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24039424, 24043520) 0 0 00000080:00200000:1.0:1789760742.242826:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.242844:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5869, 5869] 00020000:00200000:1.0:1789760742.242855:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24039424 chunk: [24039424, 24043520) 24043520, 4096 00020000:00200000:1.0:1789760742.242861:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.242882:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5869, 5869] 00000008:00010000:1.0:1789760742.242900:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.242906:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.242918:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24039424, 24043520) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.243164:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24039424=0x16ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.243228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.243231:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24043520 00000020:00010000:1.0:1789760742.243241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.243244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.243247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.243251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.243254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.243265:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.243272:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.243489:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760742.243683:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 24039424<>24043520 00000080:00200000:1.0:1789760742.292433:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5869, 5869] 00000080:00200000:1.0:1789760742.292452:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.292458:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.292473:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24043520 i_blocks 47111 00000020:00200000:1.0:1789760742.292476:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.292486:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.292521:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24043520, count: 4096 00000080:00200000:1.0:1789760742.292528:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24043520, bytes: 4096 00000080:00200000:1.0:1789760742.292539:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.292549:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.292551:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24043520, 24047616) 0 0 00000080:00200000:1.0:1789760742.292556:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.292571:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5870, 5870] 00020000:00200000:1.0:1789760742.292580:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24043520 chunk: [24043520, 24047616) 24047616, 4096 00020000:00200000:1.0:1789760742.292584:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.292668:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5870, 5870] 00000008:00010000:1.0:1789760742.292687:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.292694:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.292706:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24043520, 24047616) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.292720:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24043520=0x16ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.292779:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.292782:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24047616 00000020:00010000:1.0:1789760742.292791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.292794:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.292797:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.292800:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.292803:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.292813:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.292954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.292973:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760742.293097:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 24043520<>24047616 00000080:00200000:1.0:1789760742.334013:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5870, 5870] 00000080:00200000:1.0:1789760742.334029:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.334035:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.334052:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24047616 i_blocks 47111 00000020:00200000:1.0:1789760742.334056:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.334066:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.334112:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24047616, count: 4096 00000080:00200000:1.0:1789760742.334121:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24047616, bytes: 4096 00000080:00200000:1.0:1789760742.334133:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.334142:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.334144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24047616, 24051712) 0 0 00000080:00200000:1.0:1789760742.334149:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.334165:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5871, 5871] 00020000:00200000:1.0:1789760742.334174:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24047616 chunk: [24047616, 24051712) 24051712, 4096 00020000:00200000:1.0:1789760742.334179:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.334198:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5871, 5871] 00000008:00010000:1.0:1789760742.334215:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.334222:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.334234:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24047616, 24051712) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.334250:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24047616=0x16ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.334315:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.334319:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24051712 00000020:00010000:1.0:1789760742.334338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.334342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.334345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.334349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.334352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.334362:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.334369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.334564:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760742.334670:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 24047616<>24051712 00000080:00200000:2.0:1789760742.372627:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5871, 5871] 00000080:00200000:2.0:1789760742.372651:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.372657:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.372674:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24051712 i_blocks 47111 00000020:00200000:2.0:1789760742.372678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.372694:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.372727:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24051712, count: 4096 00000080:00200000:2.0:1789760742.372737:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24051712, bytes: 4096 00000080:00200000:2.0:1789760742.372751:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.372760:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.372762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24051712, 24055808) 0 0 00000080:00200000:2.0:1789760742.372766:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.372779:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5872, 5872] 00020000:00200000:2.0:1789760742.372786:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24051712 chunk: [24051712, 24055808) 24055808, 4096 00020000:00200000:2.0:1789760742.372789:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.372808:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5872, 5872] 00000008:00010000:2.0:1789760742.372823:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.372828:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760742.372839:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24051712, 24055808) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.372850:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24051712=0x16f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.373270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.373276:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24055808 00000020:00010000:2.0:1789760742.373290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.373293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.373296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.373300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.373303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.373313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.373320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.373360:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760742.373580:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24051712<>24055808 00000100:00100000:3.0:1789760742.373663:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493734272:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760742.415559:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973191 gen 3 00000008:00010000:2.0:1789760742.415611:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760742.415620:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760742.415633:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760742.415656:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760742.415686:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493734272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760742.472998:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5873, 5873] 00000080:00200000:2.0:1789760742.473017:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.473024:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.473042:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24059904 i_blocks 47111 00000020:00200000:2.0:1789760742.473047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.473060:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.473096:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24059904, count: 4096 00000080:00200000:2.0:1789760742.473105:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24059904, bytes: 4096 00000080:00200000:2.0:1789760742.473116:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.473125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.473128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24059904, 24064000) 0 0 00000080:00200000:2.0:1789760742.473133:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.473149:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5874, 5874] 00020000:00200000:2.0:1789760742.473158:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24059904 chunk: [24059904, 24064000) 24064000, 4096 00020000:00200000:2.0:1789760742.473163:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.473182:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5874, 5874] 00000008:00010000:2.0:1789760742.473204:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.473210:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760742.473224:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24059904, 24064000) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.473240:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24059904=0x16f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.473309:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.473312:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24064000 00000020:00010000:2.0:1789760742.473334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.473338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.473341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.473345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.473348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.473357:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.473373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.473391:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760742.473545:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24059904<>24064000 00000100:00100000:2.0:1789760742.473592:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760742.473609:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493734528:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760742.511670:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973193 gen 3 00000008:00010000:2.0:1789760742.511738:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760742.511746:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760742.511757:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760742.511952:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760742.511988:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493734528:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.513000:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493734656:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.543951:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973194 gen 3 00000008:00010000:1.0:1789760742.545064:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760742.545075:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.545087:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.545108:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760742.545138:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493734656:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.548374:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493734784:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.577249:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973195 gen 3 00000008:00010000:1.0:1789760742.577304:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760742.577313:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.577335:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.577359:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760742.577391:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493734784:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.578659:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493734912:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.619640:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973196 gen 3 00000008:00010000:1.0:1789760742.619692:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760742.619700:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.619711:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.619734:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760742.619760:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493734912:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.620430:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493735040:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760742.636634:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760742.658187:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973197 gen 3 00000008:00010000:1.0:1789760742.658242:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760742.658251:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.658262:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.658287:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760742.658317:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493735040:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.658882:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493735168:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.693198:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973198 gen 3 00000008:00010000:1.0:1789760742.693250:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760742.693258:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.693269:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.693359:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760742.693442:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493735168:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.697742:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760742.697843:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493735296:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760742.751550:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973199 gen 3 00000008:00010000:1.0:1789760742.751600:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760742.751609:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.751620:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.751977:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760742.752008:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493735296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760742.752216:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5880, 5880] 00000080:00200000:2.0:1789760742.752231:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.752238:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.752256:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24088576 i_blocks 47111 00000020:00200000:2.0:1789760742.752259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.752271:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.752305:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24088576, count: 4096 00000080:00200000:2.0:1789760742.752313:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24088576, bytes: 4096 00000080:00200000:2.0:1789760742.752334:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.752343:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.752346:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24088576, 24092672) 0 0 00000080:00200000:2.0:1789760742.752350:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.752372:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5881, 5881] 00020000:00200000:2.0:1789760742.752381:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24088576 chunk: [24088576, 24092672) 24092672, 4096 00020000:00200000:2.0:1789760742.752386:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.752431:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5881, 5881] 00000008:00010000:2.0:1789760742.752450:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.752456:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760742.752467:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24088576, 24092672) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.752482:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24088576=0x16f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.752540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.752543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24092672 00000020:00010000:2.0:1789760742.752552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.752556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.752559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.752563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.752566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.752574:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.752581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.752598:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760742.752701:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24088576<>24092672 00000100:00100000:2.0:1789760742.752751:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760742.752769:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493735424:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760742.784257:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973200 gen 3 00000008:00010000:2.0:1789760742.784308:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760742.784316:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760742.784444:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760742.784471:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760742.784501:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493735424:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760742.786215:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5881, 5881] 00000080:00200000:1.0:1789760742.786246:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.786253:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.786275:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24092672 i_blocks 47111 00000020:00200000:1.0:1789760742.786281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.786300:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.786347:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24092672, count: 4096 00000080:00200000:1.0:1789760742.786355:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24092672, bytes: 4096 00000080:00200000:1.0:1789760742.786369:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.786378:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.786380:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24092672, 24096768) 0 0 00000080:00200000:1.0:1789760742.786385:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.786439:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5882, 5882] 00020000:00200000:1.0:1789760742.786449:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24092672 chunk: [24092672, 24096768) 24096768, 4096 00020000:00200000:1.0:1789760742.786454:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.786475:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5882, 5882] 00000008:00010000:1.0:1789760742.786495:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.786501:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.786513:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24092672, 24096768) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.786527:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24092672=0x16fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.786588:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.786591:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24096768 00000020:00010000:1.0:1789760742.786598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.786602:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.786605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.786609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.786612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.786622:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.786629:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.786648:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760742.786756:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 24092672<>24096768 00000080:00200000:1.0:1789760742.831843:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5882, 5882] 00000080:00200000:1.0:1789760742.831859:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.831865:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.831882:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24096768 i_blocks 47111 00000020:00200000:1.0:1789760742.831885:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.831896:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.831931:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24096768, count: 4096 00000080:00200000:1.0:1789760742.831938:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24096768, bytes: 4096 00000080:00200000:1.0:1789760742.831948:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.831957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.831959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24096768, 24100864) 0 0 00000080:00200000:1.0:1789760742.831964:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.831982:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5883, 5883] 00020000:00200000:1.0:1789760742.831990:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24096768 chunk: [24096768, 24100864) 24100864, 4096 00020000:00200000:1.0:1789760742.831994:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.832013:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5883, 5883] 00000008:00010000:1.0:1789760742.832030:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.832037:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.832049:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24096768, 24100864) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.832062:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24096768=0x16fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.833206:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.833215:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24100864 00000020:00010000:1.0:1789760742.833228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.833232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.833236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.833241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.833244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.833259:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.833268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.833294:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760742.833444:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 24096768<>24100864 00000080:00200000:1.0:1789760742.872207:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5883, 5883] 00000080:00200000:1.0:1789760742.872224:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.872230:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.872247:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24100864 i_blocks 47111 00000020:00200000:1.0:1789760742.872251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.872261:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.872294:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24100864, count: 4096 00000080:00200000:1.0:1789760742.872301:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24100864, bytes: 4096 00000080:00200000:1.0:1789760742.872311:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.872337:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.872340:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24100864, 24104960) 0 0 00000080:00200000:1.0:1789760742.872345:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.872370:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5884, 5884] 00020000:00200000:1.0:1789760742.872378:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24100864 chunk: [24100864, 24104960) 24104960, 4096 00020000:00200000:1.0:1789760742.872387:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.872432:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5884, 5884] 00000008:00010000:1.0:1789760742.872449:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.872456:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.872468:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24100864, 24104960) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.872482:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24100864=0x16fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.872542:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.872545:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24104960 00000020:00010000:1.0:1789760742.872553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.872556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.872559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.872563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.872566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.872576:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.872583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.872598:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760742.872713:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 24100864<>24104960 00000080:00200000:1.0:1789760742.921796:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5884, 5884] 00000080:00200000:1.0:1789760742.921867:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.921874:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.921947:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24104960 i_blocks 47111 00000020:00200000:1.0:1789760742.921951:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.921960:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.921986:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24104960, count: 4096 00000080:00200000:1.0:1789760742.921993:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24104960, bytes: 4096 00000080:00200000:1.0:1789760742.922000:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.922008:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.922010:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24104960, 24109056) 0 0 00000080:00200000:1.0:1789760742.922014:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.922027:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5885, 5885] 00020000:00200000:1.0:1789760742.922034:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24104960 chunk: [24104960, 24109056) 24109056, 4096 00020000:00200000:1.0:1789760742.922038:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.922053:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5885, 5885] 00000008:00010000:1.0:1789760742.922066:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.922071:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760742.922080:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24104960, 24109056) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.922091:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24104960=0x16fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.922265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.922268:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24109056 00000020:00010000:1.0:1789760742.922275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.922278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.922280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.922283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.922599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.922609:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.922614:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.922628:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760742.922705:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 24104960<>24109056 00000080:00200000:2.0:1789760742.977502:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5885, 5885] 00000080:00200000:2.0:1789760742.977514:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.977519:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.977531:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24109056 i_blocks 47111 00000020:00200000:2.0:1789760742.977534:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.977543:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.977570:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24109056, count: 4096 00000080:00200000:2.0:1789760742.977577:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24109056, bytes: 4096 00000080:00200000:2.0:1789760742.977586:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.977593:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.977595:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24109056, 24113152) 0 0 00000080:00200000:2.0:1789760742.977598:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.977611:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5886, 5886] 00020000:00200000:2.0:1789760742.977618:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24109056 chunk: [24109056, 24113152) 24113152, 4096 00020000:00200000:2.0:1789760742.977622:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.977636:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5886, 5886] 00000008:00010000:2.0:1789760742.977649:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.977654:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760742.977662:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24109056, 24113152) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.977673:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24109056=0x16fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.977717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.977720:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24113152 00000020:00010000:2.0:1789760742.977726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.977729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.977732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.977736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.977738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.977803:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.977809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.977825:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760742.977902:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24109056<>24113152 00000100:00100000:2.0:1789760742.978509:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493736064:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760743.022130:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973205 gen 3 00000008:00010000:2.0:1789760743.022180:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760743.022187:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760743.022195:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760743.022220:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760743.022246:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493736064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760743.067529:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5887, 5887] 00000080:00200000:2.0:1789760743.067545:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.067628:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.067646:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24117248 i_blocks 47111 00000020:00200000:2.0:1789760743.067650:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.067662:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.067695:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24117248, count: 4096 00000080:00200000:2.0:1789760743.067702:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24117248, bytes: 4096 00000080:00200000:2.0:1789760743.067713:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.067722:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.067725:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24117248, 24121344) 0 0 00000080:00200000:2.0:1789760743.067730:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.067746:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5888, 5888] 00020000:00200000:2.0:1789760743.067754:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24117248 chunk: [24117248, 24121344) 24121344, 4096 00020000:00200000:2.0:1789760743.067759:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.067780:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5888, 5888] 00000008:00010000:2.0:1789760743.067918:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.067925:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760743.067938:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24117248, 24121344) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.067954:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24117248=0x1700000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.068016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.068019:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24121344 00000020:00010000:2.0:1789760743.068027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.068031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.068034:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.068038:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.068041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.068049:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.068057:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.068074:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760743.068172:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24117248<>24121344 00000100:00100000:3.0:1789760743.068251:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493736320:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760743.130246:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973207 gen 3 00000008:00010000:2.0:1789760743.130306:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760743.130316:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760743.130339:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760743.130372:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760743.130439:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493736320:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.131723:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760743.131757:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493736448:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760743.185022:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760743.185036:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.185046:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.185220:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760743.185937:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493736448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.188071:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493736576:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.229086:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973209 gen 3 00000008:00010000:1.0:1789760743.229153:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760743.229162:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.229172:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.229196:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760743.229226:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493736576:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.230077:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493736704:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760743.271426:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760743.271442:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.271453:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.271532:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760743.271565:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493736704:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760743.320072:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5892, 5892] 00000080:00200000:1.0:1789760743.320091:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.320097:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.320114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24137728 i_blocks 49159 00000020:00200000:1.0:1789760743.320119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.320130:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.320165:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24137728, count: 4096 00000080:00200000:1.0:1789760743.320171:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24137728, bytes: 4096 00000080:00200000:1.0:1789760743.320185:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.320194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.320196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24137728, 24141824) 0 0 00000080:00200000:1.0:1789760743.320201:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.320221:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5893, 5893] 00020000:00200000:1.0:1789760743.320229:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24137728 chunk: [24137728, 24141824) 24141824, 4096 00020000:00200000:1.0:1789760743.320234:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.320253:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5893, 5893] 00000008:00010000:1.0:1789760743.320272:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.320278:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760743.320291:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24137728, 24141824) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.320306:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24137728=0x1705000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.320375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.320378:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24141824 00000020:00010000:1.0:1789760743.320386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.320389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.320430:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.320435:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.320438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.320450:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.320457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.320474:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760743.323619:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 24137728<>24141824 00000100:00100000:1.0:1789760743.323727:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493736960:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.376536:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973211 gen 3 00000100:00100000:1.0:1789760743.376580:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016c700 x1876699493736960/t4294973212(4294973212) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760759 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760743.376630:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760743.376638:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.376649:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.376675:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760743.376698:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493736960:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.377912:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737088:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.413786:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973213 gen 3 00000008:00010000:1.0:1789760743.413916:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760743.413924:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.413934:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.414108:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760743.414208:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737088:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760743.444449:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5895, 5895] 00000080:00200000:1.0:1789760743.444469:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.444475:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.444493:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24150016 i_blocks 49159 00000020:00200000:1.0:1789760743.444497:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.444509:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.444545:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24150016, count: 4096 00000080:00200000:1.0:1789760743.444553:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24150016, bytes: 4096 00000080:00200000:1.0:1789760743.444568:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.444577:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.444579:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24150016, 24154112) 0 0 00000080:00200000:1.0:1789760743.444583:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.444599:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5896, 5896] 00020000:00200000:1.0:1789760743.444607:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24150016 chunk: [24150016, 24154112) 24154112, 4096 00020000:00200000:1.0:1789760743.444612:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.444631:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5896, 5896] 00000008:00010000:1.0:1789760743.445122:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.445129:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760743.445143:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24150016, 24154112) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.445159:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24150016=0x1708000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.445222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.445225:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24154112 00000020:00010000:1.0:1789760743.445233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.445236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.445239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.445243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.445246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.445255:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.445261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.445276:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760743.445518:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 24150016<>24154112 00000100:00100000:1.0:1789760743.445559:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.445578:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493737344:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.484831:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973215 gen 3 00000008:00010000:1.0:1789760743.484885:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760743.484893:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.484903:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.484926:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760743.484956:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493737344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760743.485188:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5896, 5896] 00000080:00200000:2.0:1789760743.485205:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760743.485212:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760743.485228:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24154112 i_blocks 49159 00000020:00200000:2.0:1789760743.485232:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760743.485244:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760743.485277:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24154112, count: 4096 00000080:00200000:2.0:1789760743.485285:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24154112, bytes: 4096 00000080:00200000:2.0:1789760743.485297:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760743.485306:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760743.485308:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24154112, 24158208) 0 0 00000080:00200000:2.0:1789760743.485313:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760743.486962:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5897, 5897] 00020000:00200000:2.0:1789760743.486984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24154112 chunk: [24154112, 24158208) 24158208, 4096 00020000:00200000:2.0:1789760743.486990:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760743.487091:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5897, 5897] 00000008:00010000:2.0:1789760743.487116:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760743.487122:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760743.487138:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24154112, 24158208) append 0 append lockpos 0 00000080:00200000:2.0:1789760743.487153:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24154112=0x1709000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760743.487227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760743.487230:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24158208 00000020:00010000:2.0:1789760743.487238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760743.487243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.487246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760743.487249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760743.487252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760743.487263:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760743.487270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760743.487530:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760743.487647:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24154112<>24158208 00000100:00100000:2.0:1789760743.487715:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493737472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760743.551488:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973216 gen 3 00000008:00010000:3.0:1789760743.551540:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760743.551549:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.551561:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.551592:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760743.551624:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493737472:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760743.551772:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5897, 5897] 00000080:00200000:1.0:1789760743.551790:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.551795:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.551812:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24158208 i_blocks 49159 00000020:00200000:1.0:1789760743.551815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.551826:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.551862:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24158208, count: 4096 00000080:00200000:1.0:1789760743.551870:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24158208, bytes: 4096 00000080:00200000:1.0:1789760743.552030:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.552039:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.552042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24158208, 24162304) 0 0 00000080:00200000:1.0:1789760743.552047:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.552064:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5898, 5898] 00020000:00200000:1.0:1789760743.552073:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24158208 chunk: [24158208, 24162304) 24162304, 4096 00020000:00200000:1.0:1789760743.552078:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.552112:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5898, 5898] 00000008:00010000:1.0:1789760743.552132:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.552138:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760743.552150:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24158208, 24162304) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.552164:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24158208=0x170a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.552221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.552224:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24162304 00000020:00010000:1.0:1789760743.552233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.552236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.552239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.552243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.552246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.552257:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.552263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.552280:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760743.552451:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 24158208<>24162304 00000080:00200000:1.0:1789760743.596715:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5898, 5898] 00000080:00200000:1.0:1789760743.596733:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.596739:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.596755:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24162304 i_blocks 49159 00000020:00200000:1.0:1789760743.596759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.596770:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.596875:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24162304, count: 4096 00000080:00200000:1.0:1789760743.596884:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24162304, bytes: 4096 00000080:00200000:1.0:1789760743.596896:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.596905:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.596908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24162304, 24166400) 0 0 00000080:00200000:1.0:1789760743.596912:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.596930:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5899, 5899] 00020000:00200000:1.0:1789760743.596938:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24162304 chunk: [24162304, 24166400) 24166400, 4096 00020000:00200000:1.0:1789760743.596943:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.596964:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5899, 5899] 00000008:00010000:1.0:1789760743.596983:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.596989:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760743.597002:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24162304, 24166400) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.597017:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24162304=0x170b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.597075:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.597078:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24166400 00000020:00010000:1.0:1789760743.597086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.597090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.597093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.597096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.597099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.597112:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.597119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.597136:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760743.597257:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 24162304<>24166400 00000080:00200000:1.0:1789760743.636574:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5899, 5899] 00000080:00200000:1.0:1789760743.636593:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.636600:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.636616:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24166400 i_blocks 49159 00000020:00200000:1.0:1789760743.636621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.636632:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.636665:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24166400, count: 4096 00000080:00200000:1.0:1789760743.636674:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24166400, bytes: 4096 00000080:00200000:1.0:1789760743.636688:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.636696:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.636698:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24166400, 24170496) 0 0 00000080:00200000:1.0:1789760743.636703:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.636720:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5900, 5900] 00020000:00200000:1.0:1789760743.636732:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24166400 chunk: [24166400, 24170496) 24170496, 4096 00020000:00200000:1.0:1789760743.636738:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.636757:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5900, 5900] 00000008:00010000:1.0:1789760743.636776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.636781:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760743.636792:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24166400, 24170496) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.636926:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24166400=0x170c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.636985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.636988:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24170496 00000020:00010000:1.0:1789760743.636996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.636999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.637002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.637006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac308 (ffff8eb6450b4ea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.637009:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4ea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.637019:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.637026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.637042:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760743.637268:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 24166400<>24170496 00000100:00100000:1.0:1789760743.637313:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760743.637342:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.675665:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973219 gen 3 00000008:00010000:1.0:1789760743.675725:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92cc0 00000020:00200000:1.0:1789760743.675733:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.675744:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.675807:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92cc0 00000100:00100000:1.0:1789760743.675945:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.677779:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.717827:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973220 gen 3 00000008:00010000:1.0:1789760743.717886:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760743.717895:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.717907:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.717993:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760743.718023:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493737984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.719181:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738112:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.754108:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973221 gen 3 00000008:00010000:1.0:1789760743.754163:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760743.754173:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.754185:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.754211:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760743.754246:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738112:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760743.755091:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5902, 5902] 00000080:00200000:3.0:1789760743.755144:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760743.755150:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760743.755166:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24178688 i_blocks 49159 00000020:00200000:3.0:1789760743.755170:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760743.755180:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760743.755224:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24178688, count: 4096 00000080:00200000:3.0:1789760743.755232:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24178688, bytes: 4096 00000080:00200000:3.0:1789760743.755246:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760743.755266:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760743.755268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24178688, 24182784) 0 0 00000080:00200000:3.0:1789760743.755274:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760743.755291:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5903, 5903] 00020000:00200000:3.0:1789760743.755302:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24178688 chunk: [24178688, 24182784) 24182784, 4096 00020000:00200000:3.0:1789760743.755307:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760743.755337:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5903, 5903] 00000008:00010000:3.0:1789760743.755355:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760743.755590:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760743.755606:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24178688, 24182784) append 0 append lockpos 0 00000080:00200000:3.0:1789760743.755621:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24178688=0x170f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760743.755684:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760743.755687:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24182784 00000020:00010000:3.0:1789760743.755695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760743.755698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.755701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.755704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760743.755707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760743.755716:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760743.755723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760743.755741:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760743.756194:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24178688<>24182784 00000100:00100000:2.0:1789760743.756286:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493738240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760743.798146:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973222 gen 3 00000008:00010000:3.0:1789760743.798204:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760743.798214:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.798227:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.798260:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760743.798296:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493738240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.800939:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.800978:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.846645:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973223 gen 3 00000008:00010000:1.0:1789760743.846696:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760743.846704:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.846715:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.846739:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760743.846769:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.848195:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760743.848217:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493738496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.886788:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973224 gen 3 00000008:00010000:1.0:1789760743.886849:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760743.886859:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.886871:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.886891:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760743.886921:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493738496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.887587:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.887604:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738624:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.925105:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973225 gen 3 00000008:00010000:1.0:1789760743.925160:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760743.925168:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.925179:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.925202:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760743.925232:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738624:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.926228:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760743.926247:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493738752:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760743.967300:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760743.967328:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.967340:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.967363:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760743.967439:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493738752:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760743.968502:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.968523:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.010529:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973227 gen 3 00000008:00010000:1.0:1789760744.010596:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760744.010605:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.010615:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.010646:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760744.010676:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493738880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.012830:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760744.013127:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493739008:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760744.053743:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760744.053760:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.053773:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.054895:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760744.054936:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493739008:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.057119:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760744.057146:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493739136:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.094682:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973229 gen 3 00000008:00010000:1.0:1789760744.095462:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760744.095472:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.095482:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.095505:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760744.095536:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493739136:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.096785:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760744.096817:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493739264:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760744.183833:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760744.183850:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.183861:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.183882:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760744.183917:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493739264:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.187591:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760744.187629:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493739392:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.250683:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973231 gen 3 00000008:00010000:1.0:1789760744.250756:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760744.250765:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.250777:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.250854:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760744.250886:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493739392:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760744.288504:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5913, 5913] 00000080:00200000:1.0:1789760744.288523:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.288530:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.288554:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24223744 i_blocks 49159 00000020:00200000:1.0:1789760744.288558:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.288571:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.288599:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24223744, count: 4096 00000080:00200000:1.0:1789760744.288615:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24223744, bytes: 4096 00000080:00200000:1.0:1789760744.288634:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.288648:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.288650:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24223744, 24227840) 0 0 00000080:00200000:1.0:1789760744.288654:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.288669:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5914, 5914] 00020000:00200000:1.0:1789760744.288676:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24223744 chunk: [24223744, 24227840) 24227840, 4096 00020000:00200000:1.0:1789760744.288686:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.288703:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5914, 5914] 00000008:00010000:1.0:1789760744.288717:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.288722:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760744.288730:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24223744, 24227840) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.288741:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24223744=0x171a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.288796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.288799:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24227840 00000020:00010000:1.0:1789760744.288806:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.288809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.288812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.288815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.288818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.289013:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.289020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.289039:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760744.289120:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 24223744<>24227840 00000080:00200000:1.0:1789760744.326239:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5914, 5914] 00000080:00200000:1.0:1789760744.326259:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.326266:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.326284:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24227840 i_blocks 49159 00000020:00200000:1.0:1789760744.326289:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.326302:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.326347:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24227840, count: 4096 00000080:00200000:1.0:1789760744.326354:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24227840, bytes: 4096 00000080:00200000:1.0:1789760744.326368:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.326377:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.326379:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24227840, 24231936) 0 0 00000080:00200000:1.0:1789760744.326384:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.326416:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5915, 5915] 00020000:00200000:1.0:1789760744.326425:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24227840 chunk: [24227840, 24231936) 24231936, 4096 00020000:00200000:1.0:1789760744.326430:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.326453:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5915, 5915] 00000008:00010000:1.0:1789760744.326472:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.326477:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760744.326488:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24227840, 24231936) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.326502:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24227840=0x171b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.326568:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.326571:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24231936 00000020:00010000:1.0:1789760744.326579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.326583:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.326586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.326590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.326593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.326605:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.326612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.326637:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760744.326884:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 24227840<>24231936 00000080:00200000:1.0:1789760744.391100:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5915, 5915] 00000080:00200000:1.0:1789760744.391117:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.391124:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.391139:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24231936 i_blocks 49159 00000020:00200000:1.0:1789760744.391142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.391152:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.391185:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24231936, count: 4096 00000080:00200000:1.0:1789760744.391191:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24231936, bytes: 4096 00000080:00200000:1.0:1789760744.391202:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.391211:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.391213:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24231936, 24236032) 0 0 00000080:00200000:1.0:1789760744.391217:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.391233:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5916, 5916] 00020000:00200000:1.0:1789760744.391242:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24231936 chunk: [24231936, 24236032) 24236032, 4096 00020000:00200000:1.0:1789760744.391246:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.391265:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5916, 5916] 00000008:00010000:1.0:1789760744.391282:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.391288:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760744.391301:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24231936, 24236032) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.391326:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24231936=0x171c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.391484:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.391487:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24236032 00000020:00010000:1.0:1789760744.391496:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.391500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.391503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.391508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.391511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.391521:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.391528:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.391545:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760744.391650:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 24231936<>24236032 00000080:00200000:3.0:1789760744.442626:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5916, 5916] 00000080:00200000:3.0:1789760744.442642:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.442649:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.442665:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24236032 i_blocks 49159 00000020:00200000:3.0:1789760744.442668:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.442679:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.442716:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24236032, count: 4096 00000080:00200000:3.0:1789760744.442724:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24236032, bytes: 4096 00000080:00200000:3.0:1789760744.442736:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.442745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.442747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24236032, 24240128) 0 0 00000080:00200000:3.0:1789760744.442752:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.442768:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5917, 5917] 00020000:00200000:3.0:1789760744.442776:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24236032 chunk: [24236032, 24240128) 24240128, 4096 00020000:00200000:3.0:1789760744.442780:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.442813:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5917, 5917] 00000008:00010000:3.0:1789760744.442833:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.442839:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760744.442852:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24236032, 24240128) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.442866:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24236032=0x171d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.442926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.442929:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24240128 00000020:00010000:3.0:1789760744.442938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.442941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.442944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.442948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.442951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.442959:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.442966:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.442982:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760744.443282:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24236032<>24240128 00000100:00100000:3.0:1789760744.443344:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760744.443364:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.505695:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973236 gen 3 00000008:00010000:3.0:1789760744.505743:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760744.505751:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.505762:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.505814:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760744.505852:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.505983:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5917, 5917] 00000080:00200000:2.0:1789760744.505999:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.506006:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.506023:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24240128 i_blocks 49159 00000020:00200000:2.0:1789760744.506027:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.506037:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.506070:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24240128, count: 4096 00000080:00200000:2.0:1789760744.506078:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24240128, bytes: 4096 00000080:00200000:2.0:1789760744.506088:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.506097:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.506099:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24240128, 24244224) 0 0 00000080:00200000:2.0:1789760744.506108:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.506124:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5918, 5918] 00020000:00200000:2.0:1789760744.506133:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24240128 chunk: [24240128, 24244224) 24244224, 4096 00020000:00200000:2.0:1789760744.506138:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.506157:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5918, 5918] 00000008:00010000:2.0:1789760744.506174:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.506180:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.506191:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24240128, 24244224) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.506205:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24240128=0x171e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.506264:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.506267:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24244224 00000020:00010000:2.0:1789760744.506274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.506278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.506281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.506285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.506288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.506295:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.506300:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.506326:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760744.506518:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 24240128<>24244224 00000100:00100000:3.0:1789760744.506579:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.554154:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973237 gen 3 00000008:00010000:3.0:1789760744.554203:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760744.554211:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.554222:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.554277:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760744.554304:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.555885:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5918, 5918] 00000080:00200000:2.0:1789760744.555908:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.555915:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.555932:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24244224 i_blocks 49159 00000020:00200000:2.0:1789760744.555937:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.555954:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.555988:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24244224, count: 4096 00000080:00200000:2.0:1789760744.555995:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24244224, bytes: 4096 00000080:00200000:2.0:1789760744.556010:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.556018:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.556020:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24244224, 24248320) 0 0 00000080:00200000:2.0:1789760744.556024:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.556039:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5919, 5919] 00020000:00200000:2.0:1789760744.556048:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24244224 chunk: [24244224, 24248320) 24248320, 4096 00020000:00200000:2.0:1789760744.556052:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.556084:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5919, 5919] 00000008:00010000:2.0:1789760744.556104:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.556110:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.556124:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24244224, 24248320) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.556138:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24244224=0x171f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.556205:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.556208:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24248320 00000020:00010000:2.0:1789760744.556217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.556221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.556225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.556228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.556230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.556238:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.556245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.556269:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760744.556652:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 24244224<>24248320 00000100:00100000:3.0:1789760744.556746:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493740288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.598719:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973238 gen 3 00000008:00010000:3.0:1789760744.598770:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760744.598778:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.598790:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.598869:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760744.598900:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493740288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.598987:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5919, 5919] 00000080:00200000:2.0:1789760744.599002:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.599007:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.599021:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24248320 i_blocks 49159 00000020:00200000:2.0:1789760744.599023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.599032:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.599064:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24248320, count: 4096 00000080:00200000:2.0:1789760744.599072:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24248320, bytes: 4096 00000080:00200000:2.0:1789760744.599082:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.599092:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.599094:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24248320, 24252416) 0 0 00000080:00200000:2.0:1789760744.599099:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.599114:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5920, 5920] 00020000:00200000:2.0:1789760744.599123:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24248320 chunk: [24248320, 24252416) 24252416, 4096 00020000:00200000:2.0:1789760744.599128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.599150:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5920, 5920] 00000008:00010000:2.0:1789760744.599167:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.599172:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.599183:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24248320, 24252416) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.599195:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24248320=0x1720000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.599241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.599243:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24252416 00000020:00010000:2.0:1789760744.599250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.599252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.599255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.599257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.599259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.599265:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.599270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.599283:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760744.599514:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 24248320<>24252416 00000100:00100000:2.0:1789760744.599559:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.636768:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973239 gen 3 00000008:00010000:3.0:1789760744.636821:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760744.636830:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.636842:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.636870:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760744.636901:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740416:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.637746:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493740544:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760744.682021:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760744.682037:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.682049:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.682075:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760744.682105:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493740544:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760744.682110:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5921, 5921] 00000080:00200000:3.0:1789760744.682128:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.682135:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.682150:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24256512 i_blocks 49159 00000020:00200000:3.0:1789760744.682154:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.682167:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.682197:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24256512, count: 4096 00000080:00200000:3.0:1789760744.682205:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24256512, bytes: 4096 00000080:00200000:3.0:1789760744.682215:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.682223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.682225:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24256512, 24260608) 0 0 00000080:00200000:3.0:1789760744.682230:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.682245:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5922, 5922] 00020000:00200000:3.0:1789760744.682253:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24256512 chunk: [24256512, 24260608) 24260608, 4096 00020000:00200000:3.0:1789760744.682257:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.682274:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5922, 5922] 00000008:00010000:3.0:1789760744.682292:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.682297:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760744.682320:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24256512, 24260608) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.682334:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24256512=0x1722000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.682425:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.682428:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24260608 00000020:00010000:3.0:1789760744.682436:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.682440:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.682443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.682447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.682450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.682458:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.682466:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.682483:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760744.682584:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24256512<>24260608 00000100:00100000:3.0:1789760744.682628:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760744.682644:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.723344:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973241 gen 3 00000008:00010000:3.0:1789760744.723439:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760744.723448:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.723458:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.723478:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760744.723506:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493740672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.725294:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493740800:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.752024:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973242 gen 3 00000008:00010000:1.0:1789760744.752072:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760744.752080:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.752091:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.752162:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760744.752189:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493740800:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.753111:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493740928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760744.783190:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973243 gen 3 00000008:00010000:1.0:1789760744.783242:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760744.783250:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.783261:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.783282:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760744.783323:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493740928:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.783555:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5924, 5924] 00000080:00200000:2.0:1789760744.783572:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.783578:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.783592:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24268800 i_blocks 49159 00000020:00200000:2.0:1789760744.783595:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.783606:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.783643:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24268800, count: 4096 00000080:00200000:2.0:1789760744.783651:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24268800, bytes: 4096 00000080:00200000:2.0:1789760744.783660:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.783668:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.783671:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24268800, 24272896) 0 0 00000080:00200000:2.0:1789760744.783676:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.783692:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5925, 5925] 00020000:00200000:2.0:1789760744.783699:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24268800 chunk: [24268800, 24272896) 24272896, 4096 00020000:00200000:2.0:1789760744.783704:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.783724:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5925, 5925] 00000008:00010000:2.0:1789760744.783742:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.783747:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.783759:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24268800, 24272896) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.783772:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24268800=0x1725000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.783942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.783946:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24272896 00000020:00010000:2.0:1789760744.783954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.783958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.783961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.783965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.783968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.783976:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.783983:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.784001:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760744.784109:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 24268800<>24272896 00000100:00100000:3.0:1789760744.784183:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741056:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.820251:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973244 gen 3 00000008:00010000:3.0:1789760744.820314:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760744.820323:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.820334:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.820355:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760744.820384:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741056:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.820668:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5925, 5925] 00000080:00200000:2.0:1789760744.820685:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.820692:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.820708:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24272896 i_blocks 49159 00000020:00200000:2.0:1789760744.820712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.820724:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.820760:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24272896, count: 4096 00000080:00200000:2.0:1789760744.820768:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24272896, bytes: 4096 00000080:00200000:2.0:1789760744.820776:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.820795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.820798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24272896, 24276992) 0 0 00000080:00200000:2.0:1789760744.820802:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.820975:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5926, 5926] 00020000:00200000:2.0:1789760744.820986:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24272896 chunk: [24272896, 24276992) 24276992, 4096 00020000:00200000:2.0:1789760744.820991:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.821011:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5926, 5926] 00000008:00010000:2.0:1789760744.821031:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.821038:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.821051:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24272896, 24276992) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.821064:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24272896=0x1726000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.821126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.821130:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24276992 00000020:00010000:2.0:1789760744.821139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.821142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.821146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.821149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.821152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.821160:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.821168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.821185:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760744.821289:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 24272896<>24276992 00000100:00100000:3.0:1789760744.821610:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741184:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.859317:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973245 gen 3 00000008:00010000:3.0:1789760744.859374:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760744.859382:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.859420:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.859439:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760744.859467:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741184:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.859562:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5926, 5926] 00000080:00200000:2.0:1789760744.859578:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.859584:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.859600:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24276992 i_blocks 49159 00000020:00200000:2.0:1789760744.859604:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.859614:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.859648:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24276992, count: 4096 00000080:00200000:2.0:1789760744.859655:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24276992, bytes: 4096 00000080:00200000:2.0:1789760744.859664:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.859673:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.859675:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24276992, 24281088) 0 0 00000080:00200000:2.0:1789760744.859680:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.859695:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5927, 5927] 00020000:00200000:2.0:1789760744.859703:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24276992 chunk: [24276992, 24281088) 24281088, 4096 00020000:00200000:2.0:1789760744.859708:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.859726:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5927, 5927] 00000008:00010000:2.0:1789760744.859743:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.859749:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.859761:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24276992, 24281088) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.859774:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24276992=0x1727000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.859836:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.859839:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24281088 00000020:00010000:2.0:1789760744.859847:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.859851:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.859854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.859858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.859861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.859869:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.859876:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.859892:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760744.859991:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 24276992<>24281088 00000100:00100000:3.0:1789760744.860148:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760744.860166:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741312:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.899608:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973246 gen 3 00000008:00010000:3.0:1789760744.899661:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760744.899669:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.899680:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.899705:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760744.899734:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741312:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.899887:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5927, 5927] 00000080:00200000:2.0:1789760744.899904:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.899911:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.899930:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24281088 i_blocks 49159 00000020:00200000:2.0:1789760744.899934:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.899947:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.899987:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24281088, count: 4096 00000080:00200000:2.0:1789760744.899994:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24281088, bytes: 4096 00000080:00200000:2.0:1789760744.900003:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.900011:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.900014:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24281088, 24285184) 0 0 00000080:00200000:2.0:1789760744.900018:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.900035:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5928, 5928] 00020000:00200000:2.0:1789760744.900043:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24281088 chunk: [24281088, 24285184) 24285184, 4096 00020000:00200000:2.0:1789760744.900048:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.900179:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5928, 5928] 00000008:00010000:2.0:1789760744.900198:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.900204:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.900216:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24281088, 24285184) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.900230:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24281088=0x1728000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.900284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.900287:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24285184 00000020:00010000:2.0:1789760744.900296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.900299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.900302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.900320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.900323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.900331:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.900339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.900355:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760744.900491:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24281088<>24285184 00000100:00100000:3.0:1789760744.900593:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741440:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760744.938485:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973247 gen 3 00000008:00010000:3.0:1789760744.938536:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760744.938544:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.938554:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.938614:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760744.938644:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741440:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.938732:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5928, 5928] 00000080:00200000:2.0:1789760744.938751:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.938757:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.938775:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24285184 i_blocks 49159 00000020:00200000:2.0:1789760744.938779:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.938790:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.938897:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24285184, count: 4096 00000080:00200000:2.0:1789760744.938906:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24285184, bytes: 4096 00000080:00200000:2.0:1789760744.938915:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.938922:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.938924:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24285184, 24289280) 0 0 00000080:00200000:2.0:1789760744.938928:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.938942:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5929, 5929] 00020000:00200000:2.0:1789760744.938950:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24285184 chunk: [24285184, 24289280) 24289280, 4096 00020000:00200000:2.0:1789760744.938954:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.938970:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5929, 5929] 00000008:00010000:2.0:1789760744.938986:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.938991:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.939001:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24285184, 24289280) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.939013:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24285184=0x1729000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.939064:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.939134:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24289280 00000020:00010000:2.0:1789760744.939144:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.939148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.939152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.939156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.939159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.939168:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.939174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.939188:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760744.939276:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 24285184<>24289280 00000100:00100000:2.0:1789760744.939344:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741568:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760744.975454:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760744.975468:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.975480:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.975496:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760744.975524:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760744.975529:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5929, 5929] 00000080:00200000:2.0:1789760744.975545:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.975552:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.975566:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24289280 i_blocks 49159 00000020:00200000:2.0:1789760744.975570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.975580:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.975616:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24289280, count: 4096 00000080:00200000:2.0:1789760744.975622:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24289280, bytes: 4096 00000080:00200000:2.0:1789760744.975631:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.975641:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.975643:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24289280, 24293376) 0 0 00000080:00200000:2.0:1789760744.975648:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.975664:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5930, 5930] 00020000:00200000:2.0:1789760744.975671:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24289280 chunk: [24289280, 24293376) 24293376, 4096 00020000:00200000:2.0:1789760744.975676:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.975694:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5930, 5930] 00000008:00010000:2.0:1789760744.975713:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.975719:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760744.975730:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24289280, 24293376) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.975744:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24289280=0x172a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.975802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.975805:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24293376 00000020:00010000:2.0:1789760744.975813:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.975816:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.975818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.975822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.975825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.975834:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.975841:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.975855:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760744.975959:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 24289280<>24293376 00000100:00100000:2.0:1789760744.975998:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.021205:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973249 gen 3 00000008:00010000:3.0:1789760745.021272:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760745.021282:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.021293:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.021333:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760745.022087:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493741696:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760745.022127:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:3.0:1789760745.022241:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24293376<>24297472 00000100:00100000:3.0:1789760745.022301:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.069645:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973250 gen 3 00000008:00010000:3.0:1789760745.069695:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:3.0:1789760745.069705:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.069715:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.069741:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:3.0:1789760745.069768:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493741824:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.073363:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493741952:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760745.122047:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760745.122065:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.122078:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.122103:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760745.122137:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493741952:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.124651:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760745.124682:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493742080:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.182953:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973252 gen 3 00000008:00010000:1.0:1789760745.183022:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760745.183031:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.183041:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.183064:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760745.183103:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493742080:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.185762:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760745.185791:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493742208:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760745.241926:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760745.241941:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.241952:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.241970:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760745.241997:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493742208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.242343:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5934, 5934] 00000080:00200000:2.0:1789760745.242366:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.242373:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.242388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24309760 i_blocks 49159 00000020:00200000:2.0:1789760745.242391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.242503:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.242541:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24309760, count: 4096 00000080:00200000:2.0:1789760745.242549:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24309760, bytes: 4096 00000080:00200000:2.0:1789760745.242559:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.242572:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.242574:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24309760, 24313856) 0 0 00000080:00200000:2.0:1789760745.242579:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.242596:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5935, 5935] 00020000:00200000:2.0:1789760745.242605:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24309760 chunk: [24309760, 24313856) 24313856, 4096 00020000:00200000:2.0:1789760745.242610:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.242631:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5935, 5935] 00000008:00010000:2.0:1789760745.242650:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.242655:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.242667:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24309760, 24313856) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.242681:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24309760=0x172f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.242741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.242973:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24313856 00000020:00010000:2.0:1789760745.242984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.242987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.242991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.242994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.242996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.243006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.243012:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.243029:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760745.243131:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 24309760<>24313856 00000100:00100000:3.0:1789760745.243559:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742336:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.290701:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973253 gen 3 00000100:00100000:3.0:1789760745.290745:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80000 x1876699493742336/t4294973254(4294973254) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760761 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760745.290805:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760745.290812:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.290924:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.290945:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760745.290970:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742336:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.291108:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5935, 5935] 00000080:00200000:2.0:1789760745.291124:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.291130:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.291146:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24313856 i_blocks 49159 00000020:00200000:2.0:1789760745.291149:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.291161:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.291194:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24313856, count: 4096 00000080:00200000:2.0:1789760745.291202:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24313856, bytes: 4096 00000080:00200000:2.0:1789760745.291212:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.291221:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.291223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24313856, 24317952) 0 0 00000080:00200000:2.0:1789760745.291228:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.291243:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5936, 5936] 00020000:00200000:2.0:1789760745.291252:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24313856 chunk: [24313856, 24317952) 24317952, 4096 00020000:00200000:2.0:1789760745.291334:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.291356:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5936, 5936] 00000008:00010000:2.0:1789760745.291374:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.291380:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.291392:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24313856, 24317952) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.291549:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24313856=0x1730000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.291610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.291613:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24317952 00000020:00010000:2.0:1789760745.291623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.291626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.291630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.291633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.291636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.291644:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.291651:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.291667:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760745.291773:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 24313856<>24317952 00000100:00100000:3.0:1789760745.292089:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760745.292112:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742464:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760745.328660:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973254 gen 3 00000100:00100000:2.0:1789760745.328703:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80a80 x1876699493742464/t4294973255(4294973255) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760761 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760745.328760:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:2.0:1789760745.328768:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760745.328780:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760745.330065:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:2.0:1789760745.330100:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.343598:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760745.343631:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493742592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.391101:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973256 gen 3 00000008:00010000:1.0:1789760745.391174:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760745.391183:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.391194:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.391220:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760745.391253:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493742592:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.432085:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5938, 5938] 00000080:00200000:2.0:1789760745.432100:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.432106:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.432123:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24326144 i_blocks 49159 00000020:00200000:2.0:1789760745.432127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.432139:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.432174:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24326144, count: 4096 00000080:00200000:2.0:1789760745.432182:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24326144, bytes: 4096 00000080:00200000:2.0:1789760745.432192:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.432201:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.432204:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24326144, 24330240) 0 0 00000080:00200000:2.0:1789760745.432208:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.432227:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5939, 5939] 00020000:00200000:2.0:1789760745.432235:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24326144 chunk: [24326144, 24330240) 24330240, 4096 00020000:00200000:2.0:1789760745.432240:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.432262:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5939, 5939] 00000008:00010000:2.0:1789760745.432283:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.432290:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.432315:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24326144, 24330240) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.432330:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24326144=0x1733000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.432536:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.432539:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24330240 00000020:00010000:2.0:1789760745.432549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.432552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.432555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.432558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.432561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.432569:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.432576:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.432593:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760745.432714:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 24326144<>24330240 00000100:00100000:3.0:1789760745.432826:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.478359:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973258 gen 3 00000008:00010000:3.0:1789760745.478568:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760745.478578:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.478590:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.478655:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760745.478685:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493742848:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.478763:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5939, 5939] 00000080:00200000:2.0:1789760745.478971:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.478979:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.478997:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24330240 i_blocks 49159 00000020:00200000:2.0:1789760745.479001:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.479015:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.479048:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24330240, count: 4096 00000080:00200000:2.0:1789760745.479054:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24330240, bytes: 4096 00000080:00200000:2.0:1789760745.479065:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.479074:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.479076:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24330240, 24334336) 0 0 00000080:00200000:2.0:1789760745.479080:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.479097:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5940, 5940] 00020000:00200000:2.0:1789760745.479106:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24330240 chunk: [24330240, 24334336) 24334336, 4096 00020000:00200000:2.0:1789760745.479110:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.479132:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5940, 5940] 00000008:00010000:2.0:1789760745.479150:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.479156:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.479168:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24330240, 24334336) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.479182:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24330240=0x1734000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.479245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.479248:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24334336 00000020:00010000:2.0:1789760745.479257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.479261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.479264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.479267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.479270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.479278:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.479286:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.479364:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760745.479948:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 24330240<>24334336 00000100:00100000:3.0:1789760745.480027:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493742976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.514606:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973259 gen 3 00000008:00010000:3.0:1789760745.514652:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760745.514660:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.514682:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.514700:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760745.514728:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493742976:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.514963:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5940, 5940] 00000080:00200000:2.0:1789760745.514978:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.514984:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.514999:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24334336 i_blocks 49159 00000020:00200000:2.0:1789760745.515003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.515013:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.515046:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24334336, count: 4096 00000080:00200000:2.0:1789760745.515054:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24334336, bytes: 4096 00000080:00200000:2.0:1789760745.515062:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.515071:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.515073:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24334336, 24338432) 0 0 00000080:00200000:2.0:1789760745.515078:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.515093:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5941, 5941] 00020000:00200000:2.0:1789760745.515102:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24334336 chunk: [24334336, 24338432) 24338432, 4096 00020000:00200000:2.0:1789760745.515106:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.515126:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5941, 5941] 00000008:00010000:2.0:1789760745.515141:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.515147:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.515159:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24334336, 24338432) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.515173:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24334336=0x1735000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.515231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.515234:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24338432 00000020:00010000:2.0:1789760745.515243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.515247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.515250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.515253:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.515256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.515262:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.515269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.515285:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760745.515538:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 24334336<>24338432 00000100:00100000:3.0:1789760745.515954:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743104:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.565107:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973260 gen 3 00000008:00010000:3.0:1789760745.565157:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760745.565166:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.565179:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.565200:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760745.565231:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743104:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.565336:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5941, 5941] 00000080:00200000:2.0:1789760745.565353:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.565359:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.565375:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24338432 i_blocks 49159 00000020:00200000:2.0:1789760745.565379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.565515:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.565551:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24338432, count: 4096 00000080:00200000:2.0:1789760745.565559:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24338432, bytes: 4096 00000080:00200000:2.0:1789760745.565568:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.565577:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.565580:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24338432, 24342528) 0 0 00000080:00200000:2.0:1789760745.565585:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.565601:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5942, 5942] 00020000:00200000:2.0:1789760745.565609:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24338432 chunk: [24338432, 24342528) 24342528, 4096 00020000:00200000:2.0:1789760745.565614:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.565634:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5942, 5942] 00000008:00010000:2.0:1789760745.565653:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.565659:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.565737:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24338432, 24342528) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.565753:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24338432=0x1736000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.565999:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.566002:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24342528 00000020:00010000:2.0:1789760745.566011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.566015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.566018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.566021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.566024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.566032:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.566039:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.566056:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760745.566161:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 24338432<>24342528 00000100:00100000:3.0:1789760745.566235:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493743232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.629154:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973261 gen 3 00000008:00010000:3.0:1789760745.629199:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760745.629207:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.629218:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.629237:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760745.629264:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493743232:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.629422:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5942, 5942] 00000080:00200000:2.0:1789760745.629438:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.629444:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.629459:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24342528 i_blocks 49159 00000020:00200000:2.0:1789760745.629463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.629474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.629507:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24342528, count: 4096 00000080:00200000:2.0:1789760745.629515:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24342528, bytes: 4096 00000080:00200000:2.0:1789760745.629524:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.629533:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.629535:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24342528, 24346624) 0 0 00000080:00200000:2.0:1789760745.629539:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.629554:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5943, 5943] 00020000:00200000:2.0:1789760745.629562:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24342528 chunk: [24342528, 24346624) 24346624, 4096 00020000:00200000:2.0:1789760745.629566:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.629586:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5943, 5943] 00000008:00010000:2.0:1789760745.629603:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.629609:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.629619:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24342528, 24346624) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.629633:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24342528=0x1737000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.629695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.629698:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24346624 00000020:00010000:2.0:1789760745.629706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.629710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.629713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.629717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.629720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.629730:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.629737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.629751:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760745.629863:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 24342528<>24346624 00000100:00100000:3.0:1789760745.630322:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743360:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760745.666385:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760745.666531:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.666543:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.666607:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760745.666641:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.666718:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5943, 5943] 00000080:00200000:2.0:1789760745.666736:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.666744:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.666760:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24346624 i_blocks 49159 00000020:00200000:2.0:1789760745.666763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.666774:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.666927:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24346624, count: 4096 00000080:00200000:2.0:1789760745.666935:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24346624, bytes: 4096 00000080:00200000:2.0:1789760745.666947:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.666956:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.666958:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24346624, 24350720) 0 0 00000080:00200000:2.0:1789760745.666962:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.666979:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5944, 5944] 00020000:00200000:2.0:1789760745.666987:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24346624 chunk: [24346624, 24350720) 24350720, 4096 00020000:00200000:2.0:1789760745.666992:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.667014:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5944, 5944] 00000008:00010000:2.0:1789760745.667033:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.667039:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.667051:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24346624, 24350720) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.667065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24346624=0x1738000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.667128:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.667131:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24350720 00000020:00010000:2.0:1789760745.667140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.667143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.667146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.667151:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.667154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.667163:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.667171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.667190:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760745.667318:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 24346624<>24350720 00000100:00100000:3.0:1789760745.667433:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493743488:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760745.713344:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760745.716856:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973263 gen 3 00000008:00010000:3.0:1789760745.716933:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760745.716942:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.716952:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.717015:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760745.717044:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493743488:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.717943:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5944, 5944] 00000080:00200000:2.0:1789760745.717963:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.717969:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.717985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24350720 i_blocks 49159 00000020:00200000:2.0:1789760745.717990:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.718002:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.718036:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24350720, count: 4096 00000080:00200000:2.0:1789760745.718044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24350720, bytes: 4096 00000080:00200000:2.0:1789760745.718056:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.718066:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.718068:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24350720, 24354816) 0 0 00000080:00200000:2.0:1789760745.718072:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.718089:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5945, 5945] 00020000:00200000:2.0:1789760745.718098:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24350720 chunk: [24350720, 24354816) 24354816, 4096 00020000:00200000:2.0:1789760745.718103:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.718126:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5945, 5945] 00000008:00010000:2.0:1789760745.718146:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.718152:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.718166:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24350720, 24354816) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.718182:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24350720=0x1739000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.718243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.718245:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24354816 00000020:00010000:2.0:1789760745.718254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.718257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.718260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.718264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.718266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.718275:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.718284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.718475:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760745.718593:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 24350720<>24354816 00000100:00100000:3.0:1789760745.718728:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.770999:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973264 gen 3 00000008:00010000:3.0:1789760745.771048:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760745.771056:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.771069:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.771090:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760745.771118:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743616:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.771282:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5945, 5945] 00000080:00200000:2.0:1789760745.771309:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.771317:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.771334:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24354816 i_blocks 49159 00000020:00200000:2.0:1789760745.771338:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.771349:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.771382:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24354816, count: 4096 00000080:00200000:2.0:1789760745.771390:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24354816, bytes: 4096 00000080:00200000:2.0:1789760745.771431:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.771440:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.771442:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24354816, 24358912) 0 0 00000080:00200000:2.0:1789760745.771447:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.771463:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5946, 5946] 00020000:00200000:2.0:1789760745.771471:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24354816 chunk: [24354816, 24358912) 24358912, 4096 00020000:00200000:2.0:1789760745.771475:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.771495:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5946, 5946] 00000008:00010000:2.0:1789760745.771513:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.771520:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.771531:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24354816, 24358912) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.771545:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24354816=0x173a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.771606:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.771610:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24358912 00000020:00010000:2.0:1789760745.771619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.771623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.771627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.771632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.771635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.771644:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.771652:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.771667:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760745.771787:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24354816<>24358912 00000100:00100000:3.0:1789760745.772054:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760745.772073:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743744:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760745.807599:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760745.807615:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.807627:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.807689:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760745.807719:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.807882:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5946, 5946] 00000080:00200000:2.0:1789760745.807898:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.807904:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.807919:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24358912 i_blocks 49159 00000020:00200000:2.0:1789760745.807923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.807933:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.807967:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24358912, count: 4096 00000080:00200000:2.0:1789760745.807975:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24358912, bytes: 4096 00000080:00200000:2.0:1789760745.807985:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.807993:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.807996:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24358912, 24363008) 0 0 00000080:00200000:2.0:1789760745.808000:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.808018:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5947, 5947] 00020000:00200000:2.0:1789760745.808027:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24358912 chunk: [24358912, 24363008) 24363008, 4096 00020000:00200000:2.0:1789760745.808031:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.808053:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5947, 5947] 00000008:00010000:2.0:1789760745.808071:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.808077:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.808089:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24358912, 24363008) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.808103:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24358912=0x173b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.808161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.808164:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24363008 00000020:00010000:2.0:1789760745.808173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.808176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.808180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.808183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.808186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.808193:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.808200:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.808216:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760745.808466:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 24358912<>24363008 00000100:00100000:3.0:1789760745.808525:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743872:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760745.856766:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973266 gen 3 00000008:00010000:3.0:1789760745.856835:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760745.856844:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.856856:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.856920:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760745.856952:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493743872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760745.857330:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5947, 5947] 00000080:00200000:2.0:1789760745.857348:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760745.857354:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760745.857370:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24363008 i_blocks 49159 00000020:00200000:2.0:1789760745.857374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760745.857384:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760745.857450:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24363008, count: 4096 00000080:00200000:2.0:1789760745.857457:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24363008, bytes: 4096 00000080:00200000:2.0:1789760745.857468:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760745.857477:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760745.857480:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24363008, 24367104) 0 0 00000080:00200000:2.0:1789760745.857485:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760745.857502:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5948, 5948] 00020000:00200000:2.0:1789760745.857510:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24363008 chunk: [24363008, 24367104) 24367104, 4096 00020000:00200000:2.0:1789760745.857515:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760745.857536:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5948, 5948] 00000008:00010000:2.0:1789760745.857554:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760745.857560:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760745.857572:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24363008, 24367104) append 0 append lockpos 0 00000080:00200000:2.0:1789760745.857586:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24363008=0x173c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760745.857642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760745.857645:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24367104 00000020:00010000:2.0:1789760745.857653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760745.857656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.857660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760745.857663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760745.857666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760745.857673:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760745.857679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760745.857696:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760745.857802:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 24363008<>24367104 00000100:00100000:3.0:1789760745.858325:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744000:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760745.900505:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760745.900524:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760745.900537:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760745.900617:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760745.900650:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744000:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.901467:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493744128:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760745.954870:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973268 gen 3 00000008:00010000:1.0:1789760745.954938:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760745.954946:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.954956:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.954979:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760745.955007:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493744128:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760745.955106:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5949, 5949] 00000080:00200000:3.0:1789760745.955123:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760745.955129:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760745.955144:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24371200 i_blocks 49159 00000020:00200000:3.0:1789760745.955148:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760745.955159:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760745.955191:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24371200, count: 4096 00000080:00200000:3.0:1789760745.955198:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24371200, bytes: 4096 00000080:00200000:3.0:1789760745.955207:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760745.955215:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760745.955218:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24371200, 24375296) 0 0 00000080:00200000:3.0:1789760745.955222:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760745.955237:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5950, 5950] 00020000:00200000:3.0:1789760745.955245:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24371200 chunk: [24371200, 24375296) 24375296, 4096 00020000:00200000:3.0:1789760745.955250:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760745.955268:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5950, 5950] 00000008:00010000:3.0:1789760745.955285:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760745.955291:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760745.955312:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24371200, 24375296) append 0 append lockpos 0 00000080:00200000:3.0:1789760745.955326:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24371200=0x173e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760745.955425:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760745.955428:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24375296 00000020:00010000:3.0:1789760745.955443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760745.955446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760745.955449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760745.955453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760745.955456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760745.955863:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760745.955871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760745.955892:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760745.956008:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24371200<>24375296 00000100:00100000:3.0:1789760745.956057:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760745.956087:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744256:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.016842:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973269 gen 3 00000008:00010000:3.0:1789760746.016898:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760746.016907:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.016919:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.016943:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760746.016973:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744256:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.017818:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744384:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.056292:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973270 gen 3 00000008:00010000:1.0:1789760746.056616:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760746.056627:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.056641:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.056724:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760746.056752:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744384:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.060033:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493744512:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760746.103986:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760746.104003:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.104015:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.104087:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760746.104120:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493744512:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760746.104193:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5952, 5952] 00000080:00200000:3.0:1789760746.104209:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.104215:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.104230:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24383488 i_blocks 49159 00000020:00200000:3.0:1789760746.104234:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.104244:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.104277:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24383488, count: 4096 00000080:00200000:3.0:1789760746.104285:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24383488, bytes: 4096 00000080:00200000:3.0:1789760746.104303:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.104313:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.104315:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24383488, 24387584) 0 0 00000080:00200000:3.0:1789760746.104320:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.104335:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5953, 5953] 00020000:00200000:3.0:1789760746.104344:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24383488 chunk: [24383488, 24387584) 24387584, 4096 00020000:00200000:3.0:1789760746.104429:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.104451:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5953, 5953] 00000008:00010000:3.0:1789760746.104471:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.104477:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760746.104490:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24383488, 24387584) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.104504:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24383488=0x1741000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.104560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.104563:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24387584 00000020:00010000:3.0:1789760746.104571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.104574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.104577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.104581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.104584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.104592:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.104598:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.104614:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760746.104984:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24383488<>24387584 00000100:00100000:2.0:1789760746.105070:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.134694:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973272 gen 3 00000008:00010000:3.0:1789760746.134759:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760746.134769:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.134882:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.134952:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760746.134984:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493744640:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.135748:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744768:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760746.167458:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760746.167474:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.167486:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.167498:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760746.167525:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744768:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.168281:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760746.168307:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744896:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.204326:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973274 gen 3 00000008:00010000:1.0:1789760746.204511:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760746.204521:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.204531:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.204937:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760746.204972:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493744896:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760746.205121:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5955, 5955] 00000080:00200000:3.0:1789760746.205140:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.205148:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.205166:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24395776 i_blocks 49159 00000020:00200000:3.0:1789760746.205171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.205182:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.205305:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24395776, count: 4096 00000080:00200000:3.0:1789760746.205313:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24395776, bytes: 4096 00000080:00200000:3.0:1789760746.205329:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.205338:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.205341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24395776, 24399872) 0 0 00000080:00200000:3.0:1789760746.205346:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.205471:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5956, 5956] 00020000:00200000:3.0:1789760746.205480:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24395776 chunk: [24395776, 24399872) 24399872, 4096 00020000:00200000:3.0:1789760746.205484:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.205507:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5956, 5956] 00000008:00010000:3.0:1789760746.205527:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.205533:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760746.205545:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24395776, 24399872) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.205560:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24395776=0x1744000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.205624:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.205628:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24399872 00000020:00010000:3.0:1789760746.205638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.205641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.205645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.205648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.205651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.205659:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.205666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.205685:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760746.206142:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24395776<>24399872 00000100:00100000:2.0:1789760746.206308:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493745024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.245261:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973275 gen 3 00000008:00010000:3.0:1789760746.245321:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760746.245330:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.245342:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.245372:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760746.245444:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493745024:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.246248:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493745152:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.283088:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973276 gen 3 00000008:00010000:1.0:1789760746.283138:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760746.283146:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.283158:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.283181:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760746.283212:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493745152:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.284052:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493745280:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760746.320826:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760746.320843:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.320856:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.320881:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760746.320912:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493745280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.321004:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5958, 5958] 00000080:00200000:2.0:1789760746.321020:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.321027:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.321043:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24408064 i_blocks 49159 00000020:00200000:2.0:1789760746.321047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.321057:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.321096:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24408064, count: 4096 00000080:00200000:2.0:1789760746.321104:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24408064, bytes: 4096 00000080:00200000:2.0:1789760746.321124:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.321134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.321136:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24408064, 24412160) 0 0 00000080:00200000:2.0:1789760746.321142:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.321158:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5959, 5959] 00020000:00200000:2.0:1789760746.321167:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24408064 chunk: [24408064, 24412160) 24412160, 4096 00020000:00200000:2.0:1789760746.321172:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.321192:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5959, 5959] 00000008:00010000:2.0:1789760746.321211:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.321217:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.321229:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24408064, 24412160) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.321242:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24408064=0x1747000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.321310:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.321313:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24412160 00000020:00010000:2.0:1789760746.321321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.321325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.321328:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.321332:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.321334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.321342:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.321349:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.321364:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760746.321797:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 24408064<>24412160 00000100:00100000:3.0:1789760746.321868:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493745408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.362383:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973278 gen 3 00000008:00010000:3.0:1789760746.362625:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760746.362634:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.362646:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.362674:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760746.362704:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493745408:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.363780:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493745536:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760746.396650:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760746.396668:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.396682:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.396788:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760746.396820:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493745536:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.397768:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493745664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.459240:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973280 gen 3 00000008:00010000:1.0:1789760746.459316:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760746.459324:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.459334:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.459360:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760746.459389:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493745664:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.459522:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5961, 5961] 00000080:00200000:2.0:1789760746.459539:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.459544:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.459559:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24420352 i_blocks 49159 00000020:00200000:2.0:1789760746.459562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.459572:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.459605:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24420352, count: 4096 00000080:00200000:2.0:1789760746.459612:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24420352, bytes: 4096 00000080:00200000:2.0:1789760746.459622:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.459631:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.459634:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24420352, 24424448) 0 0 00000080:00200000:2.0:1789760746.459639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.459654:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5962, 5962] 00020000:00200000:2.0:1789760746.459662:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24420352 chunk: [24420352, 24424448) 24424448, 4096 00020000:00200000:2.0:1789760746.459666:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.459686:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5962, 5962] 00000008:00010000:2.0:1789760746.459702:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.459707:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.459718:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24420352, 24424448) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.459730:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24420352=0x174a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.459797:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.459800:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24424448 00000020:00010000:2.0:1789760746.459809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.459812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.459825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.459829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.459832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.459841:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.459847:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.459863:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760746.459968:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 24420352<>24424448 00000100:00100000:3.0:1789760746.460249:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493745792:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760746.499059:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760746.499077:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.499089:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.499104:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760746.499134:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493745792:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.500136:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5962, 5962] 00000080:00200000:2.0:1789760746.500157:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.500164:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.500180:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24424448 i_blocks 49159 00000020:00200000:2.0:1789760746.500185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.500196:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.500232:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24424448, count: 4096 00000080:00200000:2.0:1789760746.500240:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24424448, bytes: 4096 00000080:00200000:2.0:1789760746.500251:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.500260:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.500262:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24424448, 24428544) 0 0 00000080:00200000:2.0:1789760746.500267:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.500283:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5963, 5963] 00020000:00200000:2.0:1789760746.500291:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24424448 chunk: [24424448, 24428544) 24428544, 4096 00020000:00200000:2.0:1789760746.500308:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.500328:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5963, 5963] 00000008:00010000:2.0:1789760746.500516:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.500523:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.500537:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24424448, 24428544) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.500551:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24424448=0x174b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.500612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.500615:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24428544 00000020:00010000:2.0:1789760746.500623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.500627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.500630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.500634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.500637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.500645:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.500652:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.500670:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760746.500900:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 24424448<>24428544 00000100:00100000:3.0:1789760746.500971:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493745920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.539656:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973281 gen 3 00000100:00100000:3.0:1789760746.539701:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81880 x1876699493745920/t4294973282(4294973282) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760762 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760746.539753:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760746.539760:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.539770:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.539898:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760746.539925:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493745920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.540017:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5963, 5963] 00000080:00200000:2.0:1789760746.540033:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.540039:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.540059:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24428544 i_blocks 49159 00000020:00200000:2.0:1789760746.540063:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.540073:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.540104:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24428544, count: 4096 00000080:00200000:2.0:1789760746.540111:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24428544, bytes: 4096 00000080:00200000:2.0:1789760746.540120:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.540128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.540130:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24428544, 24432640) 0 0 00000080:00200000:2.0:1789760746.540135:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.540150:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5964, 5964] 00020000:00200000:2.0:1789760746.540158:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24428544 chunk: [24428544, 24432640) 24432640, 4096 00020000:00200000:2.0:1789760746.540162:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.540180:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5964, 5964] 00000008:00010000:2.0:1789760746.540197:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.540203:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.540215:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24428544, 24432640) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.540227:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24428544=0x174c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.540281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.540283:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24432640 00000020:00010000:2.0:1789760746.540290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.540309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.540312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.540316:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.540319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.540327:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.540334:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.540348:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760746.540615:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 24428544<>24432640 00000100:00100000:2.0:1789760746.540667:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.581618:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973283 gen 3 00000008:00010000:3.0:1789760746.581682:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760746.581690:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.581701:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.581761:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760746.581799:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746048:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.584879:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760746.584914:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493746176:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760746.626930:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760746.626945:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.626957:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.627027:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760746.627059:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493746176:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760746.631666:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5965, 5965] 00000080:00200000:3.0:1789760746.631691:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.631698:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.631718:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24436736 i_blocks 49159 00000020:00200000:3.0:1789760746.631724:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.631743:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.631788:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24436736, count: 4096 00000080:00200000:3.0:1789760746.631798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24436736, bytes: 4096 00000080:00200000:3.0:1789760746.631815:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.631824:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.631827:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24436736, 24440832) 0 0 00000080:00200000:3.0:1789760746.631831:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.631849:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5966, 5966] 00020000:00200000:3.0:1789760746.631858:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24436736 chunk: [24436736, 24440832) 24440832, 4096 00020000:00200000:3.0:1789760746.631862:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.631885:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5966, 5966] 00000008:00010000:3.0:1789760746.631904:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.631910:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760746.631923:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24436736, 24440832) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.631938:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24436736=0x174e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.632003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.632006:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24440832 00000020:00010000:3.0:1789760746.632015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.632018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.632021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.632025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.632028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.632036:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.632043:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.632073:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760746.632180:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24436736<>24440832 00000100:00100000:3.0:1789760746.632239:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760746.632265:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.679752:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973285 gen 3 00000008:00010000:3.0:1789760746.679985:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760746.679994:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.680005:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.680100:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760746.680133:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746304:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760746.680313:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5966, 5966] 00000080:00200000:1.0:1789760746.680332:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760746.680337:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760746.680429:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24440832 i_blocks 49159 00000020:00200000:1.0:1789760746.680434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760746.680446:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760746.680481:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24440832, count: 4096 00000080:00200000:1.0:1789760746.680489:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24440832, bytes: 4096 00000080:00200000:1.0:1789760746.680500:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760746.680508:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760746.680511:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24440832, 24444928) 0 0 00000080:00200000:1.0:1789760746.680515:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760746.680533:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5967, 5967] 00020000:00200000:1.0:1789760746.680543:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24440832 chunk: [24440832, 24444928) 24444928, 4096 00020000:00200000:1.0:1789760746.680548:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760746.680569:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5967, 5967] 00000008:00010000:1.0:1789760746.680588:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760746.680594:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760746.680608:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24440832, 24444928) append 0 append lockpos 0 00000080:00200000:1.0:1789760746.680624:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24440832=0x174f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760746.680681:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760746.680685:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24444928 00000020:00010000:1.0:1789760746.680693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760746.680697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.680700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.680703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760746.680706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760746.680716:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760746.680722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760746.680740:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760746.681009:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 24440832<>24444928 00000100:00100000:1.0:1789760746.733177:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973286 gen 3 00000008:00010000:1.0:1789760746.733231:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:1.0:1789760746.733239:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.733250:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.733270:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:1.0:1789760746.733312:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493746432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.733427:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5967, 5967] 00000080:00200000:2.0:1789760746.733444:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.733450:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.733466:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24444928 i_blocks 49159 00000020:00200000:2.0:1789760746.733470:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.733480:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.733526:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24444928, count: 4096 00000080:00200000:2.0:1789760746.733533:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24444928, bytes: 4096 00000080:00200000:2.0:1789760746.733545:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.733554:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.733556:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24444928, 24449024) 0 0 00000080:00200000:2.0:1789760746.733561:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.733578:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5968, 5968] 00020000:00200000:2.0:1789760746.733587:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24444928 chunk: [24444928, 24449024) 24449024, 4096 00020000:00200000:2.0:1789760746.733592:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.733610:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5968, 5968] 00000008:00010000:2.0:1789760746.733628:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.733635:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.733646:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24444928, 24449024) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.733659:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24444928=0x1750000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.733718:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.733721:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24449024 00000020:00010000:2.0:1789760746.733730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.733733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.733736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.733739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.733743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.733751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.733759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.733920:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760746.734027:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24444928<>24449024 00000100:00100000:3.0:1789760746.734110:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746560:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760746.768985:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760746.769001:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.769013:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.769031:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760746.769062:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746560:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.769158:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5968, 5968] 00000080:00200000:2.0:1789760746.769176:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.769182:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.769198:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24449024 i_blocks 49159 00000020:00200000:2.0:1789760746.769202:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.769212:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.769245:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24449024, count: 4096 00000080:00200000:2.0:1789760746.769253:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24449024, bytes: 4096 00000080:00200000:2.0:1789760746.769262:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.769272:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.769274:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24449024, 24453120) 0 0 00000080:00200000:2.0:1789760746.769279:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.769303:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5969, 5969] 00020000:00200000:2.0:1789760746.769313:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24449024 chunk: [24449024, 24453120) 24453120, 4096 00020000:00200000:2.0:1789760746.769318:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.769337:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5969, 5969] 00000008:00010000:2.0:1789760746.769651:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.769658:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.769671:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24449024, 24453120) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.769685:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24449024=0x1751000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.769744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.769747:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24453120 00000020:00010000:2.0:1789760746.769755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.769758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.769761:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.769764:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.769767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.770289:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.770307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.770332:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760746.770497:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 24449024<>24453120 00000100:00100000:3.0:1789760746.770716:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760746.770751:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746688:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.805320:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973288 gen 3 00000008:00010000:3.0:1789760746.805390:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760746.805425:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.805435:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.805453:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760746.805480:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746688:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.805700:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5969, 5969] 00000080:00200000:2.0:1789760746.805715:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.805721:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.805735:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24453120 i_blocks 49159 00000020:00200000:2.0:1789760746.805738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.805748:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.805935:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24453120, count: 4096 00000080:00200000:2.0:1789760746.805945:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24453120, bytes: 4096 00000080:00200000:2.0:1789760746.805954:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.805963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.805965:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24453120, 24457216) 0 0 00000080:00200000:2.0:1789760746.805970:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.805985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5970, 5970] 00020000:00200000:2.0:1789760746.805994:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24453120 chunk: [24453120, 24457216) 24457216, 4096 00020000:00200000:2.0:1789760746.805998:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.806017:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5970, 5970] 00000008:00010000:2.0:1789760746.806034:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.806041:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.806053:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24453120, 24457216) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.806065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24453120=0x1752000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.806122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.806125:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24457216 00000020:00010000:2.0:1789760746.806134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.806137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.806140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.806143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.806146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.806154:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.806161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.806179:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760746.806283:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 24453120<>24457216 00000100:00100000:3.0:1789760746.806468:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746816:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.841998:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973289 gen 3 00000008:00010000:3.0:1789760746.842051:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760746.842060:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.842071:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.842089:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760746.842122:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493746816:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.842230:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5970, 5970] 00000080:00200000:2.0:1789760746.842248:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.842254:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.842271:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24457216 i_blocks 49159 00000020:00200000:2.0:1789760746.842274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.842285:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.842328:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24457216, count: 4096 00000080:00200000:2.0:1789760746.842336:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24457216, bytes: 4096 00000080:00200000:2.0:1789760746.842346:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.842356:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.842359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24457216, 24461312) 0 0 00000080:00200000:2.0:1789760746.842364:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.842384:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5971, 5971] 00020000:00200000:2.0:1789760746.842428:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24457216 chunk: [24457216, 24461312) 24461312, 4096 00020000:00200000:2.0:1789760746.842433:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.842456:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5971, 5971] 00000008:00010000:2.0:1789760746.842475:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.842480:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.842492:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24457216, 24461312) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.842507:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24457216=0x1753000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.842562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.842565:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24461312 00000020:00010000:2.0:1789760746.842574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.842578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.842581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.842585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.842588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.842595:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.842602:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.842617:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760746.842717:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 24457216<>24461312 00000100:00100000:3.0:1789760746.842790:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493746944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.877851:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973290 gen 3 00000008:00010000:3.0:1789760746.877902:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760746.877910:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.877922:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.877944:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760746.877978:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493746944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.878456:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5971, 5971] 00000080:00200000:2.0:1789760746.878477:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.878482:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.878497:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24461312 i_blocks 49159 00000020:00200000:2.0:1789760746.878501:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.878511:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.878545:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24461312, count: 4096 00000080:00200000:2.0:1789760746.878553:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24461312, bytes: 4096 00000080:00200000:2.0:1789760746.878563:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.878572:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.878574:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24461312, 24465408) 0 0 00000080:00200000:2.0:1789760746.878579:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.878596:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5972, 5972] 00020000:00200000:2.0:1789760746.878604:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24461312 chunk: [24461312, 24465408) 24465408, 4096 00020000:00200000:2.0:1789760746.878608:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.878628:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5972, 5972] 00000008:00010000:2.0:1789760746.878646:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.878652:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.878661:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24461312, 24465408) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.878674:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24461312=0x1754000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.878717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.878719:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24465408 00000020:00010000:2.0:1789760746.878726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.878730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.878733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.878736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.878739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.878745:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.878750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.878936:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:3.0:1789760746.879416:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24461312<>24465408 00000100:00100000:3.0:1789760746.879463:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760746.879484:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493747072:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760746.920566:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760746.920579:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.920590:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.920606:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760746.920630:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493747072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.921152:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5972, 5972] 00000080:00200000:2.0:1789760746.921170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.921176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.921192:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24465408 i_blocks 49159 00000020:00200000:2.0:1789760746.921197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.921208:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.921249:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24465408, count: 4096 00000080:00200000:2.0:1789760746.921256:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24465408, bytes: 4096 00000080:00200000:2.0:1789760746.921266:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.921275:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.921278:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24465408, 24469504) 0 0 00000080:00200000:2.0:1789760746.921283:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.921309:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5973, 5973] 00020000:00200000:2.0:1789760746.921317:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24465408 chunk: [24465408, 24469504) 24469504, 4096 00020000:00200000:2.0:1789760746.921321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.921341:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5973, 5973] 00000008:00010000:2.0:1789760746.921495:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.921501:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.921514:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24465408, 24469504) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.921530:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24465408=0x1755000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.921590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.921593:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24469504 00000020:00010000:2.0:1789760746.921601:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.921605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.921608:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.921611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.921614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.921622:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.921630:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.921647:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760746.921756:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 24465408<>24469504 00000100:00100000:3.0:1789760746.921960:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493747200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760746.957987:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973292 gen 3 00000008:00010000:3.0:1789760746.958056:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760746.958065:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.958077:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.958099:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760746.958130:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493747200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760746.958229:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5973, 5973] 00000080:00200000:2.0:1789760746.958248:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760746.958254:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760746.958272:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24469504 i_blocks 49159 00000020:00200000:2.0:1789760746.958276:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760746.958286:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760746.958330:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24469504, count: 4096 00000080:00200000:2.0:1789760746.958337:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24469504, bytes: 4096 00000080:00200000:2.0:1789760746.958489:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760746.958500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760746.958503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24469504, 24473600) 0 0 00000080:00200000:2.0:1789760746.958510:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760746.958528:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5974, 5974] 00020000:00200000:2.0:1789760746.958537:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24469504 chunk: [24469504, 24473600) 24473600, 4096 00020000:00200000:2.0:1789760746.958543:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760746.958563:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5974, 5974] 00000008:00010000:2.0:1789760746.958582:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760746.958587:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760746.958599:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24469504, 24473600) append 0 append lockpos 0 00000080:00200000:2.0:1789760746.958612:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24469504=0x1756000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760746.958669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760746.958671:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24473600 00000020:00010000:2.0:1789760746.958679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760746.958682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.958686:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760746.958689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760746.958692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760746.958699:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760746.958706:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760746.958722:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760746.959061:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 24469504<>24473600 00000100:00100000:3.0:1789760746.959136:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493747328:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760746.994059:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760746.994078:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.994092:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.994114:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760746.994147:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493747328:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760746.995046:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493747456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.029976:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973294 gen 3 00000008:00010000:1.0:1789760747.030054:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760747.030062:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.030072:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.030096:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760747.030126:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493747456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.034080:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493747584:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.052915:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493747840:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760747.053040:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760747.053048:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493747712:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760747.053082:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493747968:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760747.058084:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493747840:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760747.058130:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493747712:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760747.058145:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493747968:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760747.096969:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973295 gen 3 00000008:00010000:1.0:1789760747.097020:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760747.097028:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.097040:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.097065:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760747.097096:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493747584:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760747.097255:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5976, 5976] 00000080:00200000:3.0:1789760747.097272:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.097295:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.097313:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24481792 i_blocks 49159 00000020:00200000:3.0:1789760747.097317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.097328:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.097487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24481792, count: 4096 00000080:00200000:3.0:1789760747.097497:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24481792, bytes: 4096 00000080:00200000:3.0:1789760747.097507:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.097517:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.097520:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24481792, 24485888) 0 0 00000080:00200000:3.0:1789760747.097525:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.097542:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5977, 5977] 00020000:00200000:3.0:1789760747.097550:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24481792 chunk: [24481792, 24485888) 24485888, 4096 00020000:00200000:3.0:1789760747.097554:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.097574:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5977, 5977] 00000008:00010000:3.0:1789760747.097592:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.097598:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760747.097610:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24481792, 24485888) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.097624:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24481792=0x1759000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.097685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.097688:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24485888 00000020:00010000:3.0:1789760747.097697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.097700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.097703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.097707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.097710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.097718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.097725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.097741:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760747.097908:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24481792<>24485888 00000100:00100000:3.0:1789760747.097952:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760747.097970:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493748096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.134729:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973296 gen 3 00000008:00010000:3.0:1789760747.134789:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760747.134797:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.134809:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.135016:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760747.135046:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493748096:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.137064:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493748224:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.171351:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973297 gen 3 00000008:00010000:1.0:1789760747.171428:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760747.171437:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.171448:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.171520:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760747.171550:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493748224:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.172313:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748352:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.213591:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973298 gen 3 00000008:00010000:1.0:1789760747.213646:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760747.213655:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.213668:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.213690:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760747.213720:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748352:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760747.213924:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5979, 5979] 00000080:00200000:3.0:1789760747.213941:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.213947:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.213964:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24494080 i_blocks 49159 00000020:00200000:3.0:1789760747.213967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.213978:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.214013:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24494080, count: 4096 00000080:00200000:3.0:1789760747.214020:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24494080, bytes: 4096 00000080:00200000:3.0:1789760747.214029:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.214038:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.214041:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24494080, 24498176) 0 0 00000080:00200000:3.0:1789760747.214045:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.214062:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5980, 5980] 00020000:00200000:3.0:1789760747.214071:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24494080 chunk: [24494080, 24498176) 24498176, 4096 00020000:00200000:3.0:1789760747.214075:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.214095:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5980, 5980] 00000008:00010000:3.0:1789760747.214114:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.214119:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760747.214131:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24494080, 24498176) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.214145:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24494080=0x175c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.214205:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.214208:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24498176 00000020:00010000:3.0:1789760747.214216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.214220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.214223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.214227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.214229:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.214238:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.214245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.214260:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760747.214379:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24494080<>24498176 00000100:00100000:2.0:1789760747.214511:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760747.214530:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493748480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.264312:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973299 gen 3 00000008:00010000:3.0:1789760747.264366:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760747.264375:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.264386:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.264434:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760747.264463:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493748480:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760747.264572:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5980, 5980] 00000080:00200000:1.0:1789760747.264589:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.264595:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.264610:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24498176 i_blocks 49159 00000020:00200000:1.0:1789760747.264614:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.264624:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.264656:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24498176, count: 4096 00000080:00200000:1.0:1789760747.264662:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24498176, bytes: 4096 00000080:00200000:1.0:1789760747.264671:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.264681:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.264683:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24498176, 24502272) 0 0 00000080:00200000:1.0:1789760747.264688:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.264702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5981, 5981] 00020000:00200000:1.0:1789760747.264710:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24498176 chunk: [24498176, 24502272) 24502272, 4096 00020000:00200000:1.0:1789760747.264714:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.264731:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5981, 5981] 00000008:00010000:1.0:1789760747.264747:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.264752:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760747.264763:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24498176, 24502272) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.264776:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24498176=0x175d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.264834:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.264837:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24502272 00000020:00010000:1.0:1789760747.264845:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.264849:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.264852:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.264856:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.264859:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.264870:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.264876:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.264891:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760747.264987:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 24498176<>24502272 00000100:00100000:1.0:1789760747.265028:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760747.265043:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748608:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.315306:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973300 gen 3 00000008:00010000:1.0:1789760747.315357:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:1.0:1789760747.315365:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.315375:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.315436:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:1.0:1789760747.315464:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748608:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760747.315567:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5981, 5981] 00000080:00200000:3.0:1789760747.315586:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.315592:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.315607:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24502272 i_blocks 49159 00000020:00200000:3.0:1789760747.315610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.315619:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.315653:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24502272, count: 4096 00000080:00200000:3.0:1789760747.315660:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24502272, bytes: 4096 00000080:00200000:3.0:1789760747.315671:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.315679:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.315681:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24502272, 24506368) 0 0 00000080:00200000:3.0:1789760747.315686:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.315702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5982, 5982] 00020000:00200000:3.0:1789760747.315710:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24502272 chunk: [24502272, 24506368) 24506368, 4096 00020000:00200000:3.0:1789760747.315715:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.315733:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5982, 5982] 00000008:00010000:3.0:1789760747.315751:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.315757:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760747.315768:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24502272, 24506368) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.315781:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24502272=0x175e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.315928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.315931:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24506368 00000020:00010000:3.0:1789760747.315940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.315944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.315947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.315951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.315954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.315962:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.315969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.315985:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760747.316306:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24502272<>24506368 00000100:00100000:2.0:1789760747.316502:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493748736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.384630:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973301 gen 3 00000008:00010000:3.0:1789760747.384684:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760747.384694:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.384706:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.384813:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760747.384844:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493748736:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760747.384993:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5982, 5982] 00000080:00200000:1.0:1789760747.385010:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.385074:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.385092:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24506368 i_blocks 49159 00000020:00200000:1.0:1789760747.385097:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.385108:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.385140:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24506368, count: 4096 00000080:00200000:1.0:1789760747.385147:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24506368, bytes: 4096 00000080:00200000:1.0:1789760747.385158:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.385167:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.385170:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24506368, 24510464) 0 0 00000080:00200000:1.0:1789760747.385174:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.385191:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5983, 5983] 00020000:00200000:1.0:1789760747.385199:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24506368 chunk: [24506368, 24510464) 24510464, 4096 00020000:00200000:1.0:1789760747.385205:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.385223:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5983, 5983] 00000008:00010000:1.0:1789760747.385241:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.385248:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760747.385259:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24506368, 24510464) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.385273:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24506368=0x175f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.385563:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.385567:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24510464 00000020:00010000:1.0:1789760747.385577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.385581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.385585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.385588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.385591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.385604:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.385612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.385630:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760747.385733:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 24506368<>24510464 00000080:00200000:1.0:1789760747.477380:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5983, 5983] 00000080:00200000:1.0:1789760747.477417:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.477425:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.477446:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24510464 i_blocks 49159 00000020:00200000:1.0:1789760747.477452:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.477480:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.477516:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24510464, count: 4096 00000080:00200000:1.0:1789760747.477525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24510464, bytes: 4096 00000080:00200000:1.0:1789760747.477542:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.477551:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.477553:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24510464, 24514560) 0 0 00000080:00200000:1.0:1789760747.477558:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.478011:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5984, 5984] 00020000:00200000:1.0:1789760747.478079:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24510464 chunk: [24510464, 24514560) 24514560, 4096 00020000:00200000:1.0:1789760747.478085:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.478114:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5984, 5984] 00000008:00010000:1.0:1789760747.478135:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.478142:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760747.478155:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24510464, 24514560) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.478169:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24510464=0x1760000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.478237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.478240:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24514560 00000020:00010000:1.0:1789760747.478248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.478251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.478255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.478259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.478261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.478274:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.478281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.478310:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760747.478474:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 24510464<>24514560 00000100:00100000:1.0:1789760747.478516:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760747.478543:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748992:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.520374:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973303 gen 3 00000008:00010000:1.0:1789760747.520466:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760747.520476:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.520488:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.520511:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760747.520550:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493748992:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760747.558974:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5985, 5985] 00000080:00200000:3.0:1789760747.558992:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.559000:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.559018:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24518656 i_blocks 49159 00000020:00200000:3.0:1789760747.559022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.559034:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.559071:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24518656, count: 4096 00000080:00200000:3.0:1789760747.559080:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24518656, bytes: 4096 00000080:00200000:3.0:1789760747.559093:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.559102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.559105:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24518656, 24522752) 0 0 00000080:00200000:3.0:1789760747.559111:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.559129:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5986, 5986] 00020000:00200000:3.0:1789760747.559139:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24518656 chunk: [24518656, 24522752) 24522752, 4096 00020000:00200000:3.0:1789760747.559145:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.559167:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5986, 5986] 00000008:00010000:3.0:1789760747.559187:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.559193:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760747.559205:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24518656, 24522752) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.559218:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24518656=0x1762000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.559279:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.559282:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24522752 00000020:00010000:3.0:1789760747.559299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.559303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.559306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.559310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.559313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.559321:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.559328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.559643:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760747.559762:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24518656<>24522752 00000100:00100000:3.0:1789760747.559830:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760747.560025:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493749248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.603178:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973305 gen 3 00000008:00010000:3.0:1789760747.603236:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760747.603245:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.603260:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.603293:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760747.603326:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493749248:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.604963:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493749376:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760747.651421:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760747.651439:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.651452:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.651476:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760747.651509:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493749376:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.652936:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760747.653156:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493749504:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760747.703868:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973307 gen 3 00000008:00010000:1.0:1789760747.703941:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760747.703950:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.703959:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.704018:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760747.704044:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493749504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760747.704102:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5988, 5988] 00000080:00200000:3.0:1789760747.704117:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.704121:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.704133:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24530944 i_blocks 49159 00000020:00200000:3.0:1789760747.704136:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.704145:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.704174:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24530944, count: 4096 00000080:00200000:3.0:1789760747.704180:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24530944, bytes: 4096 00000080:00200000:3.0:1789760747.704190:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.704198:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.704200:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24530944, 24535040) 0 0 00000080:00200000:3.0:1789760747.704204:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.704221:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5989, 5989] 00020000:00200000:3.0:1789760747.704229:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24530944 chunk: [24530944, 24535040) 24535040, 4096 00020000:00200000:3.0:1789760747.704234:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.704255:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5989, 5989] 00000008:00010000:3.0:1789760747.704274:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.704280:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760747.704302:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24530944, 24535040) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.704322:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24530944=0x1765000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.704384:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.704387:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24535040 00000020:00010000:3.0:1789760747.705134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.705138:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.705141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.705145:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.705148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.705157:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.705165:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.705188:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760747.705307:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24530944<>24535040 00000100:00100000:2.0:1789760747.705437:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493749632:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.772317:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973308 gen 3 00000008:00010000:3.0:1789760747.772380:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760747.772389:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.772419:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.772437:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760747.772466:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493749632:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760747.773508:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5989, 5989] 00000080:00200000:1.0:1789760747.773529:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.773535:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.773552:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24535040 i_blocks 49159 00000020:00200000:1.0:1789760747.773556:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.773568:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.773602:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24535040, count: 4096 00000080:00200000:1.0:1789760747.773609:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24535040, bytes: 4096 00000080:00200000:1.0:1789760747.773621:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.773629:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.773632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24535040, 24539136) 0 0 00000080:00200000:1.0:1789760747.773636:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.773652:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5990, 5990] 00020000:00200000:1.0:1789760747.773671:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24535040 chunk: [24535040, 24539136) 24539136, 4096 00020000:00200000:1.0:1789760747.773675:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.773695:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5990, 5990] 00000008:00010000:1.0:1789760747.773713:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.773718:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760747.773729:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24535040, 24539136) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.773743:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24535040=0x1766000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.773801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.773804:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24539136 00000020:00010000:1.0:1789760747.773812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.773816:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.773819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.773822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.773825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.773836:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.773843:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.773858:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760747.773967:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 24535040<>24539136 00000080:00200000:1.0:1789760747.821705:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5990, 5990] 00000080:00200000:1.0:1789760747.821721:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760747.821727:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760747.821748:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24539136 i_blocks 49159 00000020:00200000:1.0:1789760747.821752:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760747.821762:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760747.821796:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24539136, count: 4096 00000080:00200000:1.0:1789760747.821805:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24539136, bytes: 4096 00000080:00200000:1.0:1789760747.821814:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760747.821823:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760747.821826:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24539136, 24543232) 0 0 00000080:00200000:1.0:1789760747.821830:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760747.821847:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5991, 5991] 00020000:00200000:1.0:1789760747.821855:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24539136 chunk: [24539136, 24543232) 24543232, 4096 00020000:00200000:1.0:1789760747.821859:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760747.821878:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5991, 5991] 00000008:00010000:1.0:1789760747.821896:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760747.821902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760747.821913:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24539136, 24543232) append 0 append lockpos 0 00000080:00200000:1.0:1789760747.821929:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24539136=0x1767000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760747.821985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760747.821988:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24543232 00000020:00010000:1.0:1789760747.821997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760747.822000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.822004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760747.822007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760747.822011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760747.822021:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760747.822029:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760747.822043:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760747.822143:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 24539136<>24543232 00000080:00200000:2.0:1789760747.862923:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5991, 5991] 00000080:00200000:2.0:1789760747.862941:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.862948:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.862964:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24543232 i_blocks 49159 00000020:00200000:2.0:1789760747.862968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.862978:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.863012:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24543232, count: 4096 00000080:00200000:2.0:1789760747.863021:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24543232, bytes: 4096 00000080:00200000:2.0:1789760747.863033:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.863042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.863045:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24543232, 24547328) 0 0 00000080:00200000:2.0:1789760747.863049:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.863068:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5992, 5992] 00020000:00200000:2.0:1789760747.863078:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24543232 chunk: [24543232, 24547328) 24547328, 4096 00020000:00200000:2.0:1789760747.863082:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.863102:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5992, 5992] 00000008:00010000:2.0:1789760747.863122:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.863129:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760747.863142:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24543232, 24547328) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.863159:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24543232=0x1768000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.863223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.863227:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24547328 00000020:00010000:2.0:1789760747.863236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.863239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.863243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.863246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.863249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.863258:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.863265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.863295:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760747.863447:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 24543232<>24547328 00000100:00100000:3.0:1789760747.863583:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.913131:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973311 gen 3 00000008:00010000:3.0:1789760747.913184:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760747.913194:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.913206:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.913229:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760747.913260:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750016:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760747.913389:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5992, 5992] 00000080:00200000:2.0:1789760747.913438:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.913445:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.913463:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24547328 i_blocks 49159 00000020:00200000:2.0:1789760747.913467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.913479:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.913517:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24547328, count: 4096 00000080:00200000:2.0:1789760747.913525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24547328, bytes: 4096 00000080:00200000:2.0:1789760747.913536:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.913546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.914311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24547328, 24551424) 0 0 00000080:00200000:2.0:1789760747.914320:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.914338:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5993, 5993] 00020000:00200000:2.0:1789760747.914348:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24547328 chunk: [24547328, 24551424) 24551424, 4096 00020000:00200000:2.0:1789760747.914353:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.914377:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5993, 5993] 00000008:00010000:2.0:1789760747.914429:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.914436:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760747.914449:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24547328, 24551424) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.914463:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24547328=0x1769000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.914528:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.914531:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24551424 00000020:00010000:2.0:1789760747.914540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.914544:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.914547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.915014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.915018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.915030:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.915038:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.915064:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760747.915732:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 24547328<>24551424 00000100:00100000:3.0:1789760747.916414:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760747.916446:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760747.969133:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973312 gen 3 00000008:00010000:3.0:1789760747.969185:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760747.969194:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.969207:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.969230:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760747.969260:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760747.969441:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5993, 5993] 00000080:00200000:2.0:1789760747.969458:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760747.969464:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760747.969480:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24551424 i_blocks 49159 00000020:00200000:2.0:1789760747.969483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760747.969493:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760747.969526:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24551424, count: 4096 00000080:00200000:2.0:1789760747.969533:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24551424, bytes: 4096 00000080:00200000:2.0:1789760747.969544:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760747.969552:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760747.969555:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24551424, 24555520) 0 0 00000080:00200000:2.0:1789760747.969559:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760747.969575:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5994, 5994] 00020000:00200000:2.0:1789760747.969582:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24551424 chunk: [24551424, 24555520) 24555520, 4096 00020000:00200000:2.0:1789760747.969587:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760747.969603:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5994, 5994] 00000008:00010000:2.0:1789760747.969621:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760747.969628:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760747.969639:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24551424, 24555520) append 0 append lockpos 0 00000080:00200000:2.0:1789760747.969652:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24551424=0x176a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760747.969709:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760747.969712:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24555520 00000020:00010000:2.0:1789760747.969720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760747.969723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.969726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760747.969730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760747.969733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760747.969741:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760747.969748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760747.969765:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760747.970724:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 24551424<>24555520 00000100:00100000:3.0:1789760747.971008:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760747.971034:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493750272:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760748.014254:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760748.014272:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.015568:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.015660:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760748.015696:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493750272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760748.015830:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5994, 5994] 00000080:00200000:2.0:1789760748.015849:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.015856:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.015873:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24555520 i_blocks 49159 00000020:00200000:2.0:1789760748.015878:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.015889:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.015923:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24555520, count: 4096 00000080:00200000:2.0:1789760748.015931:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24555520, bytes: 4096 00000080:00200000:2.0:1789760748.015945:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.015955:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.015957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24555520, 24559616) 0 0 00000080:00200000:2.0:1789760748.015961:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.016045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5995, 5995] 00020000:00200000:2.0:1789760748.016056:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24555520 chunk: [24555520, 24559616) 24559616, 4096 00020000:00200000:2.0:1789760748.016061:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.016082:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5995, 5995] 00000008:00010000:2.0:1789760748.016102:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.016108:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760748.016120:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24555520, 24559616) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.016138:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24555520=0x176b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.016202:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.016205:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24559616 00000020:00010000:2.0:1789760748.016215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.016218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.016222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.016225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.016228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.016237:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.016244:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.016261:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760748.016473:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 24555520<>24559616 00000100:00100000:3.0:1789760748.016536:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493750400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.045262:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973314 gen 3 00000008:00010000:3.0:1789760748.045333:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760748.045342:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.045354:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.045377:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760748.045448:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493750400:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760748.083389:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5996, 5996] 00000080:00200000:1.0:1789760748.083431:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.083437:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.083455:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24563712 i_blocks 49159 00000020:00200000:1.0:1789760748.083459:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.083471:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.083517:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24563712, count: 4096 00000080:00200000:1.0:1789760748.083525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24563712, bytes: 4096 00000080:00200000:1.0:1789760748.083537:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.083546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.083548:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24563712, 24567808) 0 0 00000080:00200000:1.0:1789760748.083552:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.083569:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5997, 5997] 00020000:00200000:1.0:1789760748.083577:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24563712 chunk: [24563712, 24567808) 24567808, 4096 00020000:00200000:1.0:1789760748.083581:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.083600:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5997, 5997] 00000008:00010000:1.0:1789760748.083618:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.083625:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.083637:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24563712, 24567808) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.083652:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24563712=0x176d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.083710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.083713:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24567808 00000020:00010000:1.0:1789760748.083721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.083725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.083729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.083732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.083735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.083746:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.083753:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.083770:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760748.083879:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 24563712<>24567808 00000100:00100000:1.0:1789760748.083923:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760748.083943:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493750656:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.131180:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973316 gen 3 00000008:00010000:1.0:1789760748.131253:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760748.131263:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.131273:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.131305:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760748.131340:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493750656:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760748.131828:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5997, 5997] 00000080:00200000:3.0:1789760748.131848:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.131855:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.131874:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24567808 i_blocks 49159 00000020:00200000:3.0:1789760748.131878:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.131890:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.131928:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24567808, count: 4096 00000080:00200000:3.0:1789760748.131936:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24567808, bytes: 4096 00000080:00200000:3.0:1789760748.131950:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.131960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.131963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24567808, 24571904) 0 0 00000080:00200000:3.0:1789760748.131968:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.131988:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5998, 5998] 00020000:00200000:3.0:1789760748.131997:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24567808 chunk: [24567808, 24571904) 24571904, 4096 00020000:00200000:3.0:1789760748.132002:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.132023:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5998, 5998] 00000008:00010000:3.0:1789760748.132043:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.132051:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760748.132063:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24567808, 24571904) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.132078:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24567808=0x176e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.132148:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.132151:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24571904 00000020:00010000:3.0:1789760748.132160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.132164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.132168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.132172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.132174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.132183:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.132190:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.132207:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760748.132324:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24567808<>24571904 00000100:00100000:2.0:1789760748.132436:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.176656:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973317 gen 3 00000008:00010000:3.0:1789760748.176705:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760748.176714:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.176724:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.176854:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760748.176884:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493750784:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760748.176997:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5998, 5998] 00000080:00200000:1.0:1789760748.177013:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.177019:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.177034:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24571904 i_blocks 49159 00000020:00200000:1.0:1789760748.177038:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.177047:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.177080:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24571904, count: 4096 00000080:00200000:1.0:1789760748.177088:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24571904, bytes: 4096 00000080:00200000:1.0:1789760748.177098:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.177107:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.177109:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24571904, 24576000) 0 0 00000080:00200000:1.0:1789760748.177113:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.177128:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [5999, 5999] 00020000:00200000:1.0:1789760748.177137:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24571904 chunk: [24571904, 24576000) 24576000, 4096 00020000:00200000:1.0:1789760748.177141:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.177158:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5999, 5999] 00000008:00010000:1.0:1789760748.177177:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.177184:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.177196:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24571904, 24576000) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.177208:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24571904=0x176f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.177431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.177435:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24576000 00000020:00010000:1.0:1789760748.177443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.177446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.177449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.177453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.177456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.177466:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.177473:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.177488:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760748.177589:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 24571904<>24576000 00000080:00200000:1.0:1789760748.222990:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [5999, 5999] 00000080:00200000:1.0:1789760748.223007:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.223013:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.223031:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24576000 i_blocks 49159 00000020:00200000:1.0:1789760748.223035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.223046:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.223081:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24576000, count: 4096 00000080:00200000:1.0:1789760748.223089:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24576000, bytes: 4096 00000080:00200000:1.0:1789760748.223107:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.223117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.223119:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24576000, 24580096) 0 0 00000080:00200000:1.0:1789760748.223124:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.223139:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6000, 6000] 00020000:00200000:1.0:1789760748.223148:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24576000 chunk: [24576000, 24580096) 24580096, 4096 00020000:00200000:1.0:1789760748.223152:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.223171:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6000, 6000] 00000008:00010000:1.0:1789760748.223189:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.223195:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.223205:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24576000, 24580096) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.223219:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24576000=0x1770000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.223276:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.223289:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24580096 00000020:00010000:1.0:1789760748.223297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.223300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.223303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.223306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.223309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.223319:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.223326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.223341:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760748.223572:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 24576000<>24580096 00000080:00200000:1.0:1789760748.291913:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6000, 6000] 00000080:00200000:1.0:1789760748.291929:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.291935:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.291951:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24580096 i_blocks 49159 00000020:00200000:1.0:1789760748.291954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.291964:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.291998:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24580096, count: 4096 00000080:00200000:1.0:1789760748.292005:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24580096, bytes: 4096 00000080:00200000:1.0:1789760748.292014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.292023:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.292025:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24580096, 24584192) 0 0 00000080:00200000:1.0:1789760748.292029:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.292045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6001, 6001] 00020000:00200000:1.0:1789760748.292052:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24580096 chunk: [24580096, 24584192) 24584192, 4096 00020000:00200000:1.0:1789760748.292057:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.292077:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6001, 6001] 00000008:00010000:1.0:1789760748.292095:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.292101:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.292113:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24580096, 24584192) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.292127:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24580096=0x1771000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.292184:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.292187:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24584192 00000020:00010000:1.0:1789760748.292196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.292199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.292202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.292206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.292209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.292220:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.292227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.292241:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760748.292354:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 24580096<>24584192 00000080:00200000:1.0:1789760748.333132:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6001, 6001] 00000080:00200000:1.0:1789760748.333149:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.333156:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.333172:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24584192 i_blocks 49159 00000020:00200000:1.0:1789760748.333176:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.333186:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.333217:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24584192, count: 4096 00000080:00200000:1.0:1789760748.333223:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24584192, bytes: 4096 00000080:00200000:1.0:1789760748.333232:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.333240:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.333242:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24584192, 24588288) 0 0 00000080:00200000:1.0:1789760748.333246:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.333261:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6002, 6002] 00020000:00200000:1.0:1789760748.333270:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24584192 chunk: [24584192, 24588288) 24588288, 4096 00020000:00200000:1.0:1789760748.333275:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.333305:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6002, 6002] 00000008:00010000:1.0:1789760748.333326:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.333332:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.333344:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24584192, 24588288) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.333359:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24584192=0x1772000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.333438:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.333441:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24588288 00000020:00010000:1.0:1789760748.333449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.333453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.333456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.333462:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.333464:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.333474:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.333481:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.333495:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760748.333588:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 24584192<>24588288 00000080:00200000:1.0:1789760748.379456:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6002, 6002] 00000080:00200000:1.0:1789760748.379478:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.379484:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.379501:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24588288 i_blocks 49159 00000020:00200000:1.0:1789760748.379506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.379519:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.379660:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24588288, count: 4096 00000080:00200000:1.0:1789760748.379669:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24588288, bytes: 4096 00000080:00200000:1.0:1789760748.379685:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.379694:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.379696:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24588288, 24592384) 0 0 00000080:00200000:1.0:1789760748.379701:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.379717:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6003, 6003] 00020000:00200000:1.0:1789760748.379725:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24588288 chunk: [24588288, 24592384) 24592384, 4096 00020000:00200000:1.0:1789760748.379730:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.379748:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6003, 6003] 00000008:00010000:1.0:1789760748.379766:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.379771:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.379783:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24588288, 24592384) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.379796:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24588288=0x1773000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.379857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.379860:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24592384 00000020:00010000:1.0:1789760748.379868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.379872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.379874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.379877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.379881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.379892:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.379899:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.379916:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760748.380021:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 24588288<>24592384 00000100:00100000:1.0:1789760748.424355:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973321 gen 3 00000100:00100000:1.0:1789760748.424415:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016d180 x1876699493751424/t4294973322(4294973322) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760764 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760748.424467:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:1.0:1789760748.424474:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.424484:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.424572:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:1.0:1789760748.424596:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493751424:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760748.424753:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6003, 6003] 00000080:00200000:3.0:1789760748.424770:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.424775:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.424789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24592384 i_blocks 49159 00000020:00200000:3.0:1789760748.424793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.424803:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.424903:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24592384, count: 4096 00000080:00200000:3.0:1789760748.424911:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24592384, bytes: 4096 00000080:00200000:3.0:1789760748.424921:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.424929:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.424932:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24592384, 24596480) 0 0 00000080:00200000:3.0:1789760748.424937:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.424953:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6004, 6004] 00020000:00200000:3.0:1789760748.424961:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24592384 chunk: [24592384, 24596480) 24596480, 4096 00020000:00200000:3.0:1789760748.424966:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.424984:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6004, 6004] 00000008:00010000:3.0:1789760748.425003:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.425010:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760748.425022:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24592384, 24596480) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.425034:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24592384=0x1774000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.425086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.425088:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24596480 00000020:00010000:3.0:1789760748.425096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.425224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.425228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.425232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.425235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.425243:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.425250:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.425266:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760748.425369:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24592384<>24596480 00000100:00100000:2.0:1789760748.425451:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493751552:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.460339:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973323 gen 3 00000008:00010000:3.0:1789760748.461777:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760748.461794:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.461807:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.461834:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760748.461868:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493751552:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760748.462496:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6004, 6004] 00000080:00200000:1.0:1789760748.462517:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.462524:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.462541:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24596480 i_blocks 49159 00000020:00200000:1.0:1789760748.462547:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.462557:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.462592:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24596480, count: 4096 00000080:00200000:1.0:1789760748.462600:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24596480, bytes: 4096 00000080:00200000:1.0:1789760748.462611:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.462620:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.462623:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24596480, 24600576) 0 0 00000080:00200000:1.0:1789760748.462627:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.462646:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6005, 6005] 00020000:00200000:1.0:1789760748.462655:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24596480 chunk: [24596480, 24600576) 24600576, 4096 00020000:00200000:1.0:1789760748.462660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.462680:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6005, 6005] 00000008:00010000:1.0:1789760748.462697:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.462703:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760748.463368:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24596480, 24600576) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.463574:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24596480=0x1775000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.463648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.463652:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24600576 00000020:00010000:1.0:1789760748.463663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.463666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.463669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.463673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.463676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.463688:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.463695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.465007:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760748.465144:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 24596480<>24600576 00000100:00100000:1.0:1789760748.465260:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760748.465296:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493751680:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760748.526729:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:1.0:1789760748.526745:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.526757:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.526783:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:1.0:1789760748.526816:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493751680:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760748.570868:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6006, 6006] 00000080:00200000:3.0:1789760748.570886:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.570892:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.570910:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24604672 i_blocks 49159 00000020:00200000:3.0:1789760748.570914:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.570925:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.570974:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24604672, count: 4096 00000080:00200000:3.0:1789760748.570982:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24604672, bytes: 4096 00000080:00200000:3.0:1789760748.570993:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.571003:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.571005:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24604672, 24608768) 0 0 00000080:00200000:3.0:1789760748.571010:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.571027:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6007, 6007] 00020000:00200000:3.0:1789760748.571036:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24604672 chunk: [24604672, 24608768) 24608768, 4096 00020000:00200000:3.0:1789760748.571041:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.571061:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6007, 6007] 00000008:00010000:3.0:1789760748.571090:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.571096:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760748.571108:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24604672, 24608768) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.571123:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24604672=0x1777000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.571183:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.571186:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24608768 00000020:00010000:3.0:1789760748.571195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.571199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.571201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.571205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.571207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.571215:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.571222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.571239:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760748.571352:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24604672<>24608768 00000100:00100000:3.0:1789760748.571443:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.571463:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493751936:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.608567:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973326 gen 3 00000008:00010000:3.0:1789760748.608617:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760748.608627:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.608636:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.608659:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760748.608689:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493751936:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.611467:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752064:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.649376:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973327 gen 3 00000008:00010000:1.0:1789760748.649449:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760748.649458:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.649470:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.649504:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760748.649537:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752064:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.650252:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493752192:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.685373:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973328 gen 3 00000008:00010000:1.0:1789760748.685475:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760748.685484:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.685496:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.685508:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760748.685538:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493752192:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.686314:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752320:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.725833:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973329 gen 3 00000008:00010000:1.0:1789760748.725886:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760748.725894:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.725905:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.725928:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760748.725959:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752320:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760748.726171:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6010, 6010] 00000080:00200000:3.0:1789760748.726187:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.726192:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.726209:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24621056 i_blocks 49159 00000020:00200000:3.0:1789760748.726214:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.726225:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.726258:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24621056, count: 4096 00000080:00200000:3.0:1789760748.726266:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24621056, bytes: 4096 00000080:00200000:3.0:1789760748.726288:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.726298:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.726300:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24621056, 24625152) 0 0 00000080:00200000:3.0:1789760748.726305:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.726323:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6011, 6011] 00020000:00200000:3.0:1789760748.726332:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24621056 chunk: [24621056, 24625152) 24625152, 4096 00020000:00200000:3.0:1789760748.726337:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.726356:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6011, 6011] 00000008:00010000:3.0:1789760748.726373:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.726379:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760748.726391:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24621056, 24625152) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.726435:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24621056=0x177b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.726494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.726497:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24625152 00000020:00010000:3.0:1789760748.726506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.726509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.726513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.726516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.726519:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.726527:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.726534:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.726549:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760748.726652:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24621056<>24625152 00000100:00100000:2.0:1789760748.726750:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760748.726778:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493752448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.765304:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973330 gen 3 00000008:00010000:3.0:1789760748.765366:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760748.765375:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.765385:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.765436:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760748.765466:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493752448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.766172:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.766190:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752576:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760748.780958:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760748.797875:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760748.797891:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.797902:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.797924:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760748.797954:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752576:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760748.798047:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6012, 6012] 00000080:00200000:3.0:1789760748.798064:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760748.798069:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760748.798085:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24629248 i_blocks 49159 00000020:00200000:3.0:1789760748.798089:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760748.798100:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760748.798132:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24629248, count: 4096 00000080:00200000:3.0:1789760748.798138:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24629248, bytes: 4096 00000080:00200000:3.0:1789760748.798149:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760748.798158:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760748.798160:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24629248, 24633344) 0 0 00000080:00200000:3.0:1789760748.798165:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760748.798180:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6013, 6013] 00020000:00200000:3.0:1789760748.798187:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24629248 chunk: [24629248, 24633344) 24633344, 4096 00020000:00200000:3.0:1789760748.798191:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760748.798210:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6013, 6013] 00000008:00010000:3.0:1789760748.798226:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760748.798233:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760748.798245:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24629248, 24633344) append 0 append lockpos 0 00000080:00200000:3.0:1789760748.798258:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24629248=0x177d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760748.798327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760748.798330:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24633344 00000020:00010000:3.0:1789760748.798339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760748.798343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.798346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760748.798349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760748.798352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760748.798360:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760748.798367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760748.798382:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760748.798610:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24629248<>24633344 00000100:00100000:2.0:1789760748.798688:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493752704:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760748.841032:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973331 gen 3 00000100:00100000:3.0:1789760748.841080:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149500 x1876699493752704/t4294973332(4294973332) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760764 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760748.841132:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760748.841140:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.841150:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.841174:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760748.841200:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493752704:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.842453:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752832:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.898954:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973332 gen 3 00000100:00100000:1.0:1789760748.899000:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304c380 x1876699493752832/t4294973333(4294973333) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760764 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760748.902230:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760748.902246:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.902259:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.902291:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760748.902326:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493752832:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.905105:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493752960:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760748.956892:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973334 gen 3 00000008:00010000:1.0:1789760748.956963:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760748.956972:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.956987:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.957010:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760748.957038:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493752960:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760749.023563:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6016, 6016] 00000080:00200000:1.0:1789760749.023582:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760749.023589:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760749.023606:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24645632 i_blocks 49159 00000020:00200000:1.0:1789760749.023610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760749.023621:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760749.024100:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24645632, count: 4096 00000080:00200000:1.0:1789760749.024111:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24645632, bytes: 4096 00000080:00200000:1.0:1789760749.024125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760749.024136:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760749.024138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24645632, 24649728) 0 0 00000080:00200000:1.0:1789760749.024142:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760749.024160:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6017, 6017] 00020000:00200000:1.0:1789760749.024168:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24645632 chunk: [24645632, 24649728) 24649728, 4096 00020000:00200000:1.0:1789760749.024172:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760749.024192:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6017, 6017] 00000008:00010000:1.0:1789760749.024211:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760749.024216:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760749.024227:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24645632, 24649728) append 0 append lockpos 0 00000080:00200000:1.0:1789760749.024242:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24645632=0x1781000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760749.025186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760749.025192:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24649728 00000020:00010000:1.0:1789760749.025203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760749.025208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.025211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.025215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760749.025222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760749.025239:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760749.025248:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760749.025284:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760749.025433:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 24645632<>24649728 00000100:00100000:1.0:1789760749.025484:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760749.025581:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493753216:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760749.076880:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973336 gen 3 00000008:00010000:1.0:1789760749.076933:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:1.0:1789760749.076942:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760749.076953:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760749.076977:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:1.0:1789760749.077011:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493753216:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760749.123883:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6018, 6018] 00000080:00200000:1.0:1789760749.123901:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760749.123907:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760749.123922:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24653824 i_blocks 49159 00000020:00200000:1.0:1789760749.123926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760749.123938:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760749.123973:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24653824, count: 4096 00000080:00200000:1.0:1789760749.123980:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24653824, bytes: 4096 00000080:00200000:1.0:1789760749.123989:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760749.125434:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760749.125439:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24653824, 24657920) 0 0 00000080:00200000:1.0:1789760749.125447:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760749.125469:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6019, 6019] 00020000:00200000:1.0:1789760749.125480:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24653824 chunk: [24653824, 24657920) 24657920, 4096 00020000:00200000:1.0:1789760749.125485:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760749.125510:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6019, 6019] 00000008:00010000:1.0:1789760749.125536:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760749.125543:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760749.125556:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24653824, 24657920) append 0 append lockpos 0 00000080:00200000:1.0:1789760749.125576:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24653824=0x1783000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760749.125643:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760749.125646:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24657920 00000020:00010000:1.0:1789760749.125655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760749.125659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.125662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.125666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760749.125669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760749.125684:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760749.125691:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760749.125716:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760749.125830:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 24653824<>24657920 00000080:00200000:2.0:1789760749.173408:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6019, 6019] 00000080:00200000:2.0:1789760749.173437:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.173444:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.173462:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24657920 i_blocks 49159 00000020:00200000:2.0:1789760749.173470:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.173489:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.173527:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24657920, count: 4096 00000080:00200000:2.0:1789760749.173536:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24657920, bytes: 4096 00000080:00200000:2.0:1789760749.173550:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.173559:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.173561:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24657920, 24662016) 0 0 00000080:00200000:2.0:1789760749.173566:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.173582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6020, 6020] 00020000:00200000:2.0:1789760749.173591:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24657920 chunk: [24657920, 24662016) 24662016, 4096 00020000:00200000:2.0:1789760749.173595:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.173617:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6020, 6020] 00000008:00010000:2.0:1789760749.173634:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.173640:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.173651:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24657920, 24662016) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.173804:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24657920=0x1784000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.173890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.173893:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24662016 00000020:00010000:2.0:1789760749.173903:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.173907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.173910:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.173914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.173917:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.173926:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.173934:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.173965:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760749.174074:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 24657920<>24662016 00000100:00100000:3.0:1789760749.174242:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493753600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760749.218737:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973339 gen 3 00000008:00010000:3.0:1789760749.218787:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760749.218796:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.218807:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.218828:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760749.218858:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493753600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760749.218967:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6020, 6020] 00000080:00200000:2.0:1789760749.218983:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.218989:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.219005:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24662016 i_blocks 49159 00000020:00200000:2.0:1789760749.219009:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.219019:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.219051:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24662016, count: 4096 00000080:00200000:2.0:1789760749.219058:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24662016, bytes: 4096 00000080:00200000:2.0:1789760749.219067:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.219076:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.219078:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24662016, 24666112) 0 0 00000080:00200000:2.0:1789760749.219083:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.219097:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6021, 6021] 00020000:00200000:2.0:1789760749.219106:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24662016 chunk: [24662016, 24666112) 24666112, 4096 00020000:00200000:2.0:1789760749.219111:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.219129:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6021, 6021] 00000008:00010000:2.0:1789760749.219145:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.219152:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.219163:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24662016, 24666112) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.219176:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24662016=0x1785000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.219237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.219240:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24666112 00000020:00010000:2.0:1789760749.219248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.219252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.219255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.219260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.219263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.219271:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.219289:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.219305:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760749.219448:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 24662016<>24666112 00000100:00100000:3.0:1789760749.219785:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493753728:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760749.263488:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973340 gen 3 00000008:00010000:3.0:1789760749.263544:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760749.263552:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.263564:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.263585:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760749.263615:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493753728:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760749.263720:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6021, 6021] 00000080:00200000:2.0:1789760749.263737:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.263744:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.263768:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24666112 i_blocks 49159 00000020:00200000:2.0:1789760749.263773:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.263785:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.263820:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24666112, count: 4096 00000080:00200000:2.0:1789760749.263827:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24666112, bytes: 4096 00000080:00200000:2.0:1789760749.263836:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.263846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.263848:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24666112, 24670208) 0 0 00000080:00200000:2.0:1789760749.263853:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.263869:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6022, 6022] 00020000:00200000:2.0:1789760749.263879:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24666112 chunk: [24666112, 24670208) 24670208, 4096 00020000:00200000:2.0:1789760749.263884:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.263903:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6022, 6022] 00000008:00010000:2.0:1789760749.263921:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.263928:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.263940:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24666112, 24670208) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.263966:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24666112=0x1786000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.264030:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.264034:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24670208 00000020:00010000:2.0:1789760749.264042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.264046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.264049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.264053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.264056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.264065:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.264072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.264088:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760749.264191:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 24666112<>24670208 00000100:00100000:3.0:1789760749.264415:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760749.264434:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493753856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760749.334123:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973341 gen 3 00000008:00010000:3.0:1789760749.334173:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760749.334182:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.334196:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.334217:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760749.334248:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493753856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760749.334360:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6022, 6022] 00000080:00200000:2.0:1789760749.334376:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.334382:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.334457:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24670208 i_blocks 49159 00000020:00200000:2.0:1789760749.334463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.334474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.334509:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24670208, count: 4096 00000080:00200000:2.0:1789760749.334518:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24670208, bytes: 4096 00000080:00200000:2.0:1789760749.334527:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.334536:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.334538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24670208, 24674304) 0 0 00000080:00200000:2.0:1789760749.334543:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.334560:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6023, 6023] 00020000:00200000:2.0:1789760749.334568:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24670208 chunk: [24670208, 24674304) 24674304, 4096 00020000:00200000:2.0:1789760749.334572:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.334591:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6023, 6023] 00000008:00010000:2.0:1789760749.334608:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.334615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.334628:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24670208, 24674304) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.334643:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24670208=0x1787000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.334705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.334708:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24674304 00000020:00010000:2.0:1789760749.334717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.334721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.334724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.334728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.334731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.334739:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.334746:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.334864:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760749.334963:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 24670208<>24674304 00000100:00100000:3.0:1789760749.335035:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.335056:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493753984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760749.390328:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973342 gen 3 00000008:00010000:3.0:1789760749.390380:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760749.390389:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.390441:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.390463:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760749.390493:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493753984:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760749.439609:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6024, 6024] 00000080:00200000:3.0:1789760749.439630:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.439641:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.440019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24678400 i_blocks 49159 00000020:00200000:3.0:1789760749.440026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.440040:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.440076:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24678400, count: 4096 00000080:00200000:3.0:1789760749.440085:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24678400, bytes: 4096 00000080:00200000:3.0:1789760749.440100:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.440109:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.440111:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24678400, 24682496) 0 0 00000080:00200000:3.0:1789760749.440116:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.440134:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6025, 6025] 00020000:00200000:3.0:1789760749.440145:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24678400 chunk: [24678400, 24682496) 24682496, 4096 00020000:00200000:3.0:1789760749.440150:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.440171:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6025, 6025] 00000008:00010000:3.0:1789760749.440189:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.440196:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760749.440208:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24678400, 24682496) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.440223:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24678400=0x1789000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.440305:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.440308:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24682496 00000020:00010000:3.0:1789760749.440321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.440324:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.440327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.440332:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.440334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.440344:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.440351:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.440372:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760749.440548:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 24678400<>24682496 00000100:00100000:2.0:1789760749.440634:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493754240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760749.513619:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973343 gen 3 00000100:00100000:3.0:1789760749.513678:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514a300 x1876699493754240/t4294973344(4294973344) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760749.513731:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760749.513739:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.513763:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.513776:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760749.513801:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493754240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760749.515049:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760749.515073:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493754368:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760749.669855:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6027, 6027] 00000080:00200000:2.0:1789760749.669872:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.669878:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.669892:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24690688 i_blocks 49159 00000020:00200000:2.0:1789760749.669896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.669907:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.669941:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24690688, count: 4096 00000080:00200000:2.0:1789760749.669948:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24690688, bytes: 4096 00000080:00200000:2.0:1789760749.669958:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.669966:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.669969:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24690688, 24694784) 0 0 00000080:00200000:2.0:1789760749.669973:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.670000:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6028, 6028] 00020000:00200000:2.0:1789760749.670009:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24690688 chunk: [24690688, 24694784) 24694784, 4096 00020000:00200000:2.0:1789760749.670013:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.670032:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6028, 6028] 00000008:00010000:2.0:1789760749.670049:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.670055:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.670067:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24690688, 24694784) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.670082:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24690688=0x178c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.670162:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.670165:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24694784 00000020:00010000:2.0:1789760749.670174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.670177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.670180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.670184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.670187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.670196:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.670202:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.670218:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760749.670323:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24690688<>24694784 00000100:00100000:3.0:1789760749.670542:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493754624:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760749.722934:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973347 gen 3 00000008:00010000:2.0:1789760749.722987:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760749.722996:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.723008:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.723032:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760749.723062:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493754624:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760749.805052:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6029, 6029] 00000080:00200000:2.0:1789760749.805069:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.805075:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.805091:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24698880 i_blocks 49159 00000020:00200000:2.0:1789760749.805094:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.805103:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.805137:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24698880, count: 4096 00000080:00200000:2.0:1789760749.805144:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24698880, bytes: 4096 00000080:00200000:2.0:1789760749.805155:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.805164:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.805166:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24698880, 24702976) 0 0 00000080:00200000:2.0:1789760749.805171:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.805188:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6030, 6030] 00020000:00200000:2.0:1789760749.805197:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24698880 chunk: [24698880, 24702976) 24702976, 4096 00020000:00200000:2.0:1789760749.805202:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.805220:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6030, 6030] 00000008:00010000:2.0:1789760749.805238:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.805244:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760749.805255:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24698880, 24702976) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.805280:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24698880=0x178e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.805334:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.805337:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24702976 00000020:00010000:2.0:1789760749.805345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.805349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.805351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.805356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.805358:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.805366:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.805373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.805388:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760749.805528:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24698880<>24702976 00000100:00100000:2.0:1789760749.810475:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.810518:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493754880:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760749.847974:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973349 gen 3 00000008:00010000:2.0:1789760749.848027:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760749.848036:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.848049:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.848082:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760749.848113:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493754880:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760749.848323:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6030, 6030] 00000080:00200000:3.0:1789760749.848349:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.848355:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.848372:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24702976 i_blocks 49159 00000020:00200000:3.0:1789760749.848376:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.848388:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.848454:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24702976, count: 4096 00000080:00200000:3.0:1789760749.848463:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24702976, bytes: 4096 00000080:00200000:3.0:1789760749.848478:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.848487:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.848490:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24702976, 24707072) 0 0 00000080:00200000:3.0:1789760749.848494:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.848512:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6031, 6031] 00020000:00200000:3.0:1789760749.848520:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24702976 chunk: [24702976, 24707072) 24707072, 4096 00020000:00200000:3.0:1789760749.848525:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.848719:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6031, 6031] 00000008:00010000:3.0:1789760749.848740:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.848746:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760749.848759:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24702976, 24707072) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.848773:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24702976=0x178f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.849034:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.849038:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24707072 00000020:00010000:3.0:1789760749.849046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.849050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.849053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.849057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.849060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.849131:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.849139:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.849158:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760749.849263:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 24702976<>24707072 00000100:00100000:3.0:1789760749.849324:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.849906:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493755008:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760749.884755:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760749.884770:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.884781:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.884804:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760749.884833:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493755008:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760749.884855:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6031, 6031] 00000080:00200000:3.0:1789760749.884871:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.884877:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.884891:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24707072 i_blocks 49159 00000020:00200000:3.0:1789760749.884895:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.884904:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.884935:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24707072, count: 4096 00000080:00200000:3.0:1789760749.884941:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24707072, bytes: 4096 00000080:00200000:3.0:1789760749.884953:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.884961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.884963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24707072, 24711168) 0 0 00000080:00200000:3.0:1789760749.884968:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.884983:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6032, 6032] 00020000:00200000:3.0:1789760749.884991:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24707072 chunk: [24707072, 24711168) 24711168, 4096 00020000:00200000:3.0:1789760749.884995:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.885014:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6032, 6032] 00000008:00010000:3.0:1789760749.885032:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.885038:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760749.885051:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24707072, 24711168) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.885065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24707072=0x1790000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.885135:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.885138:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24711168 00000020:00010000:3.0:1789760749.885147:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.885150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.885154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.885157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.885160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.885167:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.885175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.885190:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760749.885316:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 24707072<>24711168 00000100:00100000:2.0:1789760749.885633:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.885653:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755136:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760749.926989:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973351 gen 3 00000008:00010000:2.0:1789760749.927049:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760749.927057:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.927067:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.927086:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760749.927112:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755136:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760749.927236:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6032, 6032] 00000080:00200000:3.0:1789760749.927250:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.927256:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.927280:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24711168 i_blocks 49159 00000020:00200000:3.0:1789760749.927284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.927294:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.927389:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24711168, count: 4096 00000080:00200000:3.0:1789760749.927434:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24711168, bytes: 4096 00000080:00200000:3.0:1789760749.927445:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.927453:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.927456:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24711168, 24715264) 0 0 00000080:00200000:3.0:1789760749.927461:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.927476:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6033, 6033] 00020000:00200000:3.0:1789760749.927484:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24711168 chunk: [24711168, 24715264) 24715264, 4096 00020000:00200000:3.0:1789760749.927489:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.927508:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6033, 6033] 00000008:00010000:3.0:1789760749.927525:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.927532:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760749.927543:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24711168, 24715264) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.927556:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24711168=0x1791000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.927609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.927611:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24715264 00000020:00010000:3.0:1789760749.927618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.927622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.927626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.927630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.927633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.927641:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.927648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.927727:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760749.928023:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 24711168<>24715264 00000100:00100000:2.0:1789760749.928094:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755264:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760749.970745:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760749.970969:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760749.970981:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760749.971003:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760749.971033:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755264:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760749.971191:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6033, 6033] 00000080:00200000:3.0:1789760749.971206:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.971211:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.971226:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24715264 i_blocks 49159 00000020:00200000:3.0:1789760749.971231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.971240:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.971281:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24715264, count: 4096 00000080:00200000:3.0:1789760749.971289:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24715264, bytes: 4096 00000080:00200000:3.0:1789760749.971298:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.971307:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.971309:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24715264, 24719360) 0 0 00000080:00200000:3.0:1789760749.971313:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.971329:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6034, 6034] 00020000:00200000:3.0:1789760749.971337:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24715264 chunk: [24715264, 24719360) 24719360, 4096 00020000:00200000:3.0:1789760749.971342:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.971484:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6034, 6034] 00000008:00010000:3.0:1789760749.971504:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.971509:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760749.971521:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24715264, 24719360) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.971534:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24715264=0x1792000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.971587:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.971589:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24719360 00000020:00010000:3.0:1789760749.971598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.971602:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.971605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.971609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.971612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.971618:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.971626:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.971642:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760749.972166:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 24715264<>24719360 00000100:00100000:2.0:1789760749.972427:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.972449:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755392:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.005119:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973353 gen 3 00000008:00010000:2.0:1789760750.005185:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760750.005195:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.005204:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.005224:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760750.005255:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755392:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760750.005478:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6034, 6034] 00000080:00200000:3.0:1789760750.005496:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.005503:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.005520:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24719360 i_blocks 49159 00000020:00200000:3.0:1789760750.005523:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.005533:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.005566:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24719360, count: 4096 00000080:00200000:3.0:1789760750.005573:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24719360, bytes: 4096 00000080:00200000:3.0:1789760750.005583:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.005592:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.005594:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24719360, 24723456) 0 0 00000080:00200000:3.0:1789760750.005599:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.005616:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6035, 6035] 00020000:00200000:3.0:1789760750.005624:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24719360 chunk: [24719360, 24723456) 24723456, 4096 00020000:00200000:3.0:1789760750.005629:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.005649:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6035, 6035] 00000008:00010000:3.0:1789760750.005678:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.005685:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760750.005697:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24719360, 24723456) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.005713:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24719360=0x1793000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.005901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.005904:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24723456 00000020:00010000:3.0:1789760750.005913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.005916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.005920:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.005924:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.005927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.005935:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.005954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.005970:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760750.006076:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 24719360<>24723456 00000100:00100000:2.0:1789760750.006341:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755520:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.043000:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973354 gen 3 00000008:00010000:2.0:1789760750.043057:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760750.043065:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.043078:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.043101:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760750.043130:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755520:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760750.043243:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6035, 6035] 00000080:00200000:3.0:1789760750.043810:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.043819:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.043977:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24723456 i_blocks 49159 00000020:00200000:2.0:1789760750.043983:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.043994:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.044027:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24723456, count: 4096 00000080:00200000:2.0:1789760750.044033:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24723456, bytes: 4096 00000080:00200000:2.0:1789760750.044043:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.044050:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.044052:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24723456, 24727552) 0 0 00000080:00200000:2.0:1789760750.044057:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.044072:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6036, 6036] 00020000:00200000:2.0:1789760750.044080:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24723456 chunk: [24723456, 24727552) 24727552, 4096 00020000:00200000:2.0:1789760750.044085:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.044104:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6036, 6036] 00000008:00010000:2.0:1789760750.044120:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.044126:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.044137:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24723456, 24727552) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.044151:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24723456=0x1794000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.044209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.044212:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24727552 00000020:00010000:2.0:1789760750.044220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.044223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.044226:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.044229:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.044232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.044240:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.044247:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.044262:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.044554:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24723456<>24727552 00000100:00100000:2.0:1789760750.044616:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760750.044637:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755648:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.078779:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973355 gen 3 00000008:00010000:2.0:1789760750.078892:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760750.078900:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.078912:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.078926:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760750.078962:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493755648:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760750.080024:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760750.080046:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493755776:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760750.176840:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973356 gen 3 00000008:00010000:1.0:1789760750.176895:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760750.176904:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.176916:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.176938:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760750.176969:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493755776:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760750.238331:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6038, 6038] 00000080:00200000:1.0:1789760750.238354:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.238361:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.238388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24735744 i_blocks 49159 00000020:00200000:1.0:1789760750.238419:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.238435:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.238468:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24735744, count: 4096 00000080:00200000:1.0:1789760750.238476:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24735744, bytes: 4096 00000080:00200000:1.0:1789760750.238490:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.238498:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.238500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24735744, 24739840) 0 0 00000080:00200000:1.0:1789760750.238505:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.238521:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6039, 6039] 00020000:00200000:1.0:1789760750.238530:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24735744 chunk: [24735744, 24739840) 24739840, 4096 00020000:00200000:1.0:1789760750.238534:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.238552:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6039, 6039] 00000008:00010000:1.0:1789760750.238569:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.238575:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.238586:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24735744, 24739840) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.238600:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24735744=0x1797000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.238674:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.238678:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24739840 00000020:00010000:1.0:1789760750.238686:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.238691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.238694:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.238697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.238700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.238712:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.238718:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.238735:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760750.238845:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 24735744<>24739840 00000100:00100000:1.0:1789760750.238900:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760750.238923:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493756032:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760750.294154:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760750.294173:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.294185:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.294217:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760750.294248:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493756032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760750.294366:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6039, 6039] 00000080:00200000:2.0:1789760750.294384:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.294429:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.294445:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24739840 i_blocks 49159 00000020:00200000:2.0:1789760750.294449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.294461:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.294499:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24739840, count: 4096 00000080:00200000:2.0:1789760750.294507:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24739840, bytes: 4096 00000080:00200000:2.0:1789760750.294517:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.294526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.294528:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24739840, 24743936) 0 0 00000080:00200000:2.0:1789760750.294533:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.294548:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6040, 6040] 00020000:00200000:2.0:1789760750.294558:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24739840 chunk: [24739840, 24743936) 24743936, 4096 00020000:00200000:2.0:1789760750.294563:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.294583:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6040, 6040] 00000008:00010000:2.0:1789760750.294602:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.294609:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.294621:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24739840, 24743936) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.294637:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24739840=0x1798000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.294700:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.294703:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24743936 00000020:00010000:2.0:1789760750.294713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.294717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.294719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.294723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.294725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.294734:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.294741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.294878:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.294977:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24739840<>24743936 00000100:00100000:3.0:1789760750.295136:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756160:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.345433:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973359 gen 3 00000008:00010000:2.0:1789760750.345635:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760750.345646:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.345659:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.345750:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760750.345782:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756160:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760750.347389:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6040, 6040] 00000080:00200000:1.0:1789760750.347424:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.347431:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.347448:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24743936 i_blocks 49159 00000020:00200000:1.0:1789760750.347454:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.347468:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.348662:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24743936, count: 4096 00000080:00200000:1.0:1789760750.348678:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24743936, bytes: 4096 00000080:00200000:1.0:1789760750.348692:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.348701:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.348704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24743936, 24748032) 0 0 00000080:00200000:1.0:1789760750.348709:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.348726:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6041, 6041] 00020000:00200000:1.0:1789760750.348734:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24743936 chunk: [24743936, 24748032) 24748032, 4096 00020000:00200000:1.0:1789760750.348739:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.348759:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6041, 6041] 00000008:00010000:1.0:1789760750.348776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.348782:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.348792:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24743936, 24748032) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.348806:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24743936=0x1799000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.348865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.348868:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24748032 00000020:00010000:1.0:1789760750.348875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.348878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.348882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.348886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.348888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.348901:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.348908:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.348932:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760750.349040:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 24743936<>24748032 00000080:00200000:1.0:1789760750.403606:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6041, 6041] 00000080:00200000:1.0:1789760750.403632:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.403639:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.403658:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24748032 i_blocks 49159 00000020:00200000:1.0:1789760750.403664:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.403680:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.403714:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24748032, count: 4096 00000080:00200000:1.0:1789760750.403723:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24748032, bytes: 4096 00000080:00200000:1.0:1789760750.403739:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.403748:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.403750:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24748032, 24752128) 0 0 00000080:00200000:1.0:1789760750.403755:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.403773:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6042, 6042] 00020000:00200000:1.0:1789760750.403783:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24748032 chunk: [24748032, 24752128) 24752128, 4096 00020000:00200000:1.0:1789760750.403788:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.403810:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6042, 6042] 00000008:00010000:1.0:1789760750.403829:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.403836:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.403849:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24748032, 24752128) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.403865:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24748032=0x179a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.403933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.403936:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24752128 00000020:00010000:1.0:1789760750.403945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.403949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.403952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.403955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.403958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.403970:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.403977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.403995:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760750.404108:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 24748032<>24752128 00000080:00200000:1.0:1789760750.471172:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6042, 6042] 00000080:00200000:1.0:1789760750.471189:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.471195:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.471212:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24752128 i_blocks 49159 00000020:00200000:1.0:1789760750.471216:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.471227:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.471260:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24752128, count: 4096 00000080:00200000:1.0:1789760750.471278:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24752128, bytes: 4096 00000080:00200000:1.0:1789760750.471288:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.471297:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.471299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24752128, 24756224) 0 0 00000080:00200000:1.0:1789760750.471304:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.471320:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6043, 6043] 00020000:00200000:1.0:1789760750.471328:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24752128 chunk: [24752128, 24756224) 24756224, 4096 00020000:00200000:1.0:1789760750.471332:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.471352:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6043, 6043] 00000008:00010000:1.0:1789760750.471379:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.471384:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.471428:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24752128, 24756224) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.471442:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24752128=0x179b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.471500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.471503:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24756224 00000020:00010000:1.0:1789760750.471511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.471515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.471518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.471522:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.471525:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.471535:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.471544:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.471559:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760750.471659:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 24752128<>24756224 00000100:00100000:1.0:1789760750.471707:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760750.471723:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493756544:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760750.534618:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973362 gen 3 00000008:00010000:1.0:1789760750.534672:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760750.534682:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.534692:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.534717:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760750.534746:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493756544:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760750.535001:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6043, 6043] 00000080:00200000:2.0:1789760750.535019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.535025:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.535040:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24756224 i_blocks 49159 00000020:00200000:2.0:1789760750.535045:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.535055:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.535086:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24756224, count: 4096 00000080:00200000:2.0:1789760750.535094:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24756224, bytes: 4096 00000080:00200000:2.0:1789760750.535105:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.535114:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.535117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24756224, 24760320) 0 0 00000080:00200000:2.0:1789760750.535122:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.535136:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6044, 6044] 00020000:00200000:2.0:1789760750.535145:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24756224 chunk: [24756224, 24760320) 24760320, 4096 00020000:00200000:2.0:1789760750.535149:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.535169:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6044, 6044] 00000008:00010000:2.0:1789760750.535189:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.535196:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.535208:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24756224, 24760320) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.535221:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24756224=0x179c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.535284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.535287:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24760320 00000020:00010000:2.0:1789760750.535295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.535298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.535301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.535305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6430dd308 (ffff8eb650f97cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.535307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb650f97cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.535315:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.535322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.535334:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.535594:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24756224<>24760320 00000100:00100000:2.0:1789760750.535632:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760750.535650:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756672:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.590635:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973363 gen 3 00000008:00010000:2.0:1789760750.590688:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760750.590696:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.590708:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.590798:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760750.590828:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756672:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760750.593599:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6044, 6044] 00000080:00200000:3.0:1789760750.593628:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.593635:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.593655:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24760320 i_blocks 49159 00000020:00200000:3.0:1789760750.593660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.593678:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.593745:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24760320, count: 4096 00000080:00200000:3.0:1789760750.593755:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24760320, bytes: 4096 00000080:00200000:3.0:1789760750.593770:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.593779:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.593782:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24760320, 24764416) 0 0 00000080:00200000:3.0:1789760750.593787:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.593802:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6045, 6045] 00020000:00200000:3.0:1789760750.593811:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24760320 chunk: [24760320, 24764416) 24764416, 4096 00020000:00200000:3.0:1789760750.593815:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.593836:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6045, 6045] 00000008:00010000:3.0:1789760750.593853:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.593860:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760750.593872:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24760320, 24764416) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.593889:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24760320=0x179d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.593957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.593961:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24764416 00000020:00010000:3.0:1789760750.593970:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.593973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.593977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.593980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.593983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.593990:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.593998:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.594023:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760750.594132:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 24760320<>24764416 00000100:00100000:2.0:1789760750.594232:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756800:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.636514:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973364 gen 3 00000008:00010000:2.0:1789760750.636567:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760750.636577:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.636730:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.636868:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760750.636900:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493756800:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760750.636980:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6045, 6045] 00000080:00200000:3.0:1789760750.636997:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.637002:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.637018:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24764416 i_blocks 49159 00000020:00200000:3.0:1789760750.637021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.637031:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.637072:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24764416, count: 4096 00000080:00200000:3.0:1789760750.637084:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24764416, bytes: 4096 00000080:00200000:3.0:1789760750.637094:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.637103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.637105:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24764416, 24768512) 0 0 00000080:00200000:3.0:1789760750.637109:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.637125:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6046, 6046] 00020000:00200000:3.0:1789760750.637133:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24764416 chunk: [24764416, 24768512) 24768512, 4096 00020000:00200000:3.0:1789760750.637138:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.637157:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6046, 6046] 00000008:00010000:3.0:1789760750.637175:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.637182:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760750.637193:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24764416, 24768512) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.637239:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24764416=0x179e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.637312:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.637357:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24768512 00000020:00010000:3.0:1789760750.637367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.637371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.637374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.637378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.637382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.637391:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.637494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.637511:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760750.637675:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 24764416<>24768512 00000100:00100000:2.0:1789760750.637758:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493756928:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760750.685001:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760750.685018:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.685031:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.685061:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760750.685095:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493756928:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760750.685122:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6046, 6046] 00000080:00200000:1.0:1789760750.685138:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.685145:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.685160:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24768512 i_blocks 49159 00000020:00200000:1.0:1789760750.685165:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.685177:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.685211:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24768512, count: 4096 00000080:00200000:1.0:1789760750.685219:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24768512, bytes: 4096 00000080:00200000:1.0:1789760750.685234:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.685243:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.685245:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24768512, 24772608) 0 0 00000080:00200000:1.0:1789760750.685250:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.685279:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6047, 6047] 00020000:00200000:1.0:1789760750.685289:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24768512 chunk: [24768512, 24772608) 24772608, 4096 00020000:00200000:1.0:1789760750.685294:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.685315:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6047, 6047] 00000008:00010000:1.0:1789760750.685908:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.685916:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.685929:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24768512, 24772608) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.685945:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24768512=0x179f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.686008:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.686011:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24772608 00000020:00010000:1.0:1789760750.686019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.686023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.686026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.686030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.686032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.686045:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.686052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.686068:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760750.686181:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 24768512<>24772608 00000080:00200000:2.0:1789760750.733302:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6047, 6047] 00000080:00200000:2.0:1789760750.733319:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.733325:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.733340:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24772608 i_blocks 49159 00000020:00200000:2.0:1789760750.733344:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.733355:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.733390:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24772608, count: 4096 00000080:00200000:2.0:1789760750.733439:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24772608, bytes: 4096 00000080:00200000:2.0:1789760750.733449:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.733458:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.733460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24772608, 24776704) 0 0 00000080:00200000:2.0:1789760750.733465:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.733482:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6048, 6048] 00020000:00200000:2.0:1789760750.733491:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24772608 chunk: [24772608, 24776704) 24776704, 4096 00020000:00200000:2.0:1789760750.733497:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.733517:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6048, 6048] 00000008:00010000:2.0:1789760750.733535:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.733541:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.733553:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24772608, 24776704) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.733568:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24772608=0x17a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.733625:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.733628:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24776704 00000020:00010000:2.0:1789760750.733637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.733640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.733644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.733648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.733651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.733659:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.733666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.733689:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.733805:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24772608<>24776704 00000100:00100000:2.0:1789760750.733852:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760750.733870:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493757184:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.789170:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973367 gen 3 00000008:00010000:2.0:1789760750.789231:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760750.789239:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.789252:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.789288:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760750.789318:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493757184:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760750.863230:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760750.863247:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.863258:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.863292:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760750.863324:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493757312:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760750.863556:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6049, 6049] 00000080:00200000:2.0:1789760750.863572:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.863577:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.863594:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24780800 i_blocks 49159 00000020:00200000:2.0:1789760750.863599:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.863619:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.863655:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24780800, count: 4096 00000080:00200000:2.0:1789760750.863662:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24780800, bytes: 4096 00000080:00200000:2.0:1789760750.863673:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.863682:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.863684:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24780800, 24784896) 0 0 00000080:00200000:2.0:1789760750.863689:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.863705:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6050, 6050] 00020000:00200000:2.0:1789760750.863714:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24780800 chunk: [24780800, 24784896) 24784896, 4096 00020000:00200000:2.0:1789760750.863718:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.863738:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6050, 6050] 00000008:00010000:2.0:1789760750.863756:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.863762:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.863772:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24780800, 24784896) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.863787:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24780800=0x17a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.863848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.863851:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24784896 00000020:00010000:2.0:1789760750.863859:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.863863:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.863866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.863869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.863872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.863880:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.863886:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.863903:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.864557:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 24780800<>24784896 00000100:00100000:3.0:1789760750.864655:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493757440:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760750.916647:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973368 gen 3 00000100:00100000:2.0:1789760750.916699:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83b80 x1876699493757440/t4294973369(4294973369) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760766 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760750.916749:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760750.916758:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.916769:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.916798:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760750.916832:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493757440:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760750.916833:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6050, 6050] 00000080:00200000:1.0:1789760750.916851:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.916857:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.916872:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24784896 i_blocks 49159 00000020:00200000:1.0:1789760750.916875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.916886:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.916921:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24784896, count: 4096 00000080:00200000:1.0:1789760750.916928:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24784896, bytes: 4096 00000080:00200000:1.0:1789760750.916952:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.916961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.916963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24784896, 24788992) 0 0 00000080:00200000:1.0:1789760750.916968:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.916985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6051, 6051] 00020000:00200000:1.0:1789760750.916993:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24784896 chunk: [24784896, 24788992) 24788992, 4096 00020000:00200000:1.0:1789760750.916998:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.917018:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6051, 6051] 00000008:00010000:1.0:1789760750.917037:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.917043:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760750.917056:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24784896, 24788992) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.917071:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24784896=0x17a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.917132:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.917135:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24788992 00000020:00010000:1.0:1789760750.917143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.917147:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.917150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.917153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.917156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.917167:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.917174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.917189:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760750.917311:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 24784896<>24788992 00000100:00100000:1.0:1789760750.917359:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760750.917377:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493757568:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760750.969171:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973370 gen 3 00000008:00010000:1.0:1789760750.969243:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:1.0:1789760750.969253:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.969272:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.969300:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:1.0:1789760750.969332:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493757568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760750.969446:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6051, 6051] 00000080:00200000:2.0:1789760750.969463:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.969469:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.969486:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24788992 i_blocks 49159 00000020:00200000:2.0:1789760750.969490:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.969501:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.969535:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24788992, count: 4096 00000080:00200000:2.0:1789760750.969543:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24788992, bytes: 4096 00000080:00200000:2.0:1789760750.969554:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.969563:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.969566:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24788992, 24793088) 0 0 00000080:00200000:2.0:1789760750.969570:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.969587:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6052, 6052] 00020000:00200000:2.0:1789760750.969597:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24788992 chunk: [24788992, 24793088) 24793088, 4096 00020000:00200000:2.0:1789760750.969602:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.969620:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6052, 6052] 00000008:00010000:2.0:1789760750.969637:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.969644:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760750.969656:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24788992, 24793088) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.969670:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24788992=0x17a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.969730:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.969734:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24793088 00000020:00010000:2.0:1789760750.969742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.969746:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.969749:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.969753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.969757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.969765:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.969838:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.969853:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760750.969971:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 24788992<>24793088 00000100:00100000:3.0:1789760750.970060:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493757696:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760751.025929:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973371 gen 3 00000008:00010000:2.0:1789760751.025987:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760751.025995:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.026008:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.026087:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760751.026119:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493757696:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760751.027541:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6052, 6052] 00000080:00200000:1.0:1789760751.027562:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.027569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.027589:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24793088 i_blocks 49159 00000020:00200000:1.0:1789760751.027594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.027607:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.027639:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24793088, count: 4096 00000080:00200000:1.0:1789760751.027647:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24793088, bytes: 4096 00000080:00200000:1.0:1789760751.027659:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.027668:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.027670:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24793088, 24797184) 0 0 00000080:00200000:1.0:1789760751.027674:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.027691:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6053, 6053] 00020000:00200000:1.0:1789760751.027699:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24793088 chunk: [24793088, 24797184) 24797184, 4096 00020000:00200000:1.0:1789760751.027703:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.027723:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6053, 6053] 00000008:00010000:1.0:1789760751.027742:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.027748:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760751.027760:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24793088, 24797184) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.027777:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24793088=0x17a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.028129:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.028179:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24797184 00000020:00010000:1.0:1789760751.028254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.028267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.028270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.028274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.028277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.028288:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.028295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.028312:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760751.031868:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 24793088<>24797184 00000080:00200000:1.0:1789760751.119137:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6053, 6053] 00000080:00200000:1.0:1789760751.119154:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.119160:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.119176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24797184 i_blocks 49159 00000020:00200000:1.0:1789760751.119180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.119190:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.119225:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24797184, count: 4096 00000080:00200000:1.0:1789760751.119232:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24797184, bytes: 4096 00000080:00200000:1.0:1789760751.119241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.119250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.119253:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24797184, 24801280) 0 0 00000080:00200000:1.0:1789760751.119267:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.119285:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6054, 6054] 00020000:00200000:1.0:1789760751.119294:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24797184 chunk: [24797184, 24801280) 24801280, 4096 00020000:00200000:1.0:1789760751.119299:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.119330:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6054, 6054] 00000008:00010000:1.0:1789760751.119359:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.119366:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760751.119379:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24797184, 24801280) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.119435:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24797184=0x17a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.119495:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.119498:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24801280 00000020:00010000:1.0:1789760751.119508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.119511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.119514:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.119517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.119520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.119531:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.119539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.119554:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760751.119657:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 24797184<>24801280 00000080:00200000:1.0:1789760751.192162:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6054, 6054] 00000080:00200000:1.0:1789760751.192179:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.192185:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.192200:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24801280 i_blocks 49159 00000020:00200000:1.0:1789760751.192204:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.192214:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.192246:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24801280, count: 4096 00000080:00200000:1.0:1789760751.192253:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24801280, bytes: 4096 00000080:00200000:1.0:1789760751.192272:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.192281:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.192284:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24801280, 24805376) 0 0 00000080:00200000:1.0:1789760751.192289:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.192306:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6055, 6055] 00020000:00200000:1.0:1789760751.192380:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24801280 chunk: [24801280, 24805376) 24805376, 4096 00020000:00200000:1.0:1789760751.192386:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.192433:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6055, 6055] 00000008:00010000:1.0:1789760751.192453:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.192459:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760751.192471:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24801280, 24805376) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.192485:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24801280=0x17a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.192543:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.192546:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24805376 00000020:00010000:1.0:1789760751.192556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.192559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.192562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.192566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.192569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.192579:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.192586:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.192601:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760751.192887:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 24801280<>24805376 00000100:00100000:1.0:1789760751.192931:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760751.192951:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493758080:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.245048:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973374 gen 3 00000008:00010000:1.0:1789760751.245115:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:1.0:1789760751.245123:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.245132:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.245157:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:1.0:1789760751.245184:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493758080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760751.279473:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6056, 6056] 00000080:00200000:2.0:1789760751.279490:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.279495:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.279513:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24809472 i_blocks 49159 00000020:00200000:2.0:1789760751.279518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.279541:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.279573:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24809472, count: 4096 00000080:00200000:2.0:1789760751.279580:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24809472, bytes: 4096 00000080:00200000:2.0:1789760751.279590:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.279599:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.279601:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24809472, 24813568) 0 0 00000080:00200000:2.0:1789760751.279605:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.279620:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6057, 6057] 00020000:00200000:2.0:1789760751.279629:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24809472 chunk: [24809472, 24813568) 24813568, 4096 00020000:00200000:2.0:1789760751.279633:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.279652:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6057, 6057] 00000008:00010000:2.0:1789760751.279669:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.279675:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760751.279687:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24809472, 24813568) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.279701:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24809472=0x17a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.279932:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.279935:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24813568 00000020:00010000:2.0:1789760751.279945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.279948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.279951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.279955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.279958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.279965:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.279972:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.279988:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760751.280092:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 24809472<>24813568 00000100:00100000:2.0:1789760751.280139:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760751.280157:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493758336:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760751.322359:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973376 gen 3 00000008:00010000:2.0:1789760751.322440:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:2.0:1789760751.322449:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.322460:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.322485:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:2.0:1789760751.322517:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493758336:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.323373:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493758464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.401291:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973377 gen 3 00000008:00010000:1.0:1789760751.401343:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760751.401352:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.401363:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.401385:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760751.401442:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493758464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.402169:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493758592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.451338:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973378 gen 3 00000008:00010000:1.0:1789760751.451390:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760751.451452:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.451464:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.451557:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6059, 6059] 00000080:00200000:3.0:1789760751.451575:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.451581:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.451596:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24821760 i_blocks 49159 00000020:00200000:3.0:1789760751.451601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.451613:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.451646:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24821760, count: 4096 00000080:00200000:3.0:1789760751.451654:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24821760, bytes: 4096 00000080:00200000:3.0:1789760751.451665:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.451674:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.451677:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24821760, 24825856) 0 0 00000080:00200000:3.0:1789760751.451682:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.451699:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6060, 6060] 00020000:00200000:3.0:1789760751.451707:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24821760 chunk: [24821760, 24825856) 24825856, 4096 00020000:00200000:3.0:1789760751.451711:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.451747:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6060, 6060] 00000008:00010000:3.0:1789760751.451765:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.451772:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.451784:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24821760, 24825856) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.451798:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24821760=0x17ac000, pages 1 (max 131072), lockless, parallel 00000008:00010000:1.0:1789760751.451915:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:3.0:1789760751.452858:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.452864:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24825856 00000020:00010000:3.0:1789760751.452876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.452889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.452893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.452898:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.452901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.452912:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.452919:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.452942:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760751.453050:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 24821760<>24825856 00000100:00100000:3.0:1789760751.453098:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760751.453125:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.455467:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493758592:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760751.497971:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973379 gen 3 00000008:00010000:2.0:1789760751.498029:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760751.498038:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.498049:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.498070:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760751.498103:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758720:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760751.498210:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6060, 6060] 00000080:00200000:3.0:1789760751.498228:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.498234:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.498249:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24825856 i_blocks 49159 00000020:00200000:3.0:1789760751.498253:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.498274:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.498322:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24825856, count: 4096 00000080:00200000:3.0:1789760751.498329:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24825856, bytes: 4096 00000080:00200000:3.0:1789760751.498339:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.498348:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.498350:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24825856, 24829952) 0 0 00000080:00200000:3.0:1789760751.498355:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.498371:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6061, 6061] 00020000:00200000:3.0:1789760751.498379:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24825856 chunk: [24825856, 24829952) 24829952, 4096 00020000:00200000:3.0:1789760751.498383:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.498431:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6061, 6061] 00000008:00010000:3.0:1789760751.498451:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.498457:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.498469:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24825856, 24829952) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.498482:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24825856=0x17ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.498539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.498541:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24829952 00000020:00010000:3.0:1789760751.498549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.498553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.498556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.498560:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.498562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.498570:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.498577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.498592:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760751.498694:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 24825856<>24829952 00000100:00100000:2.0:1789760751.499582:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758848:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760751.554612:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760751.554629:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.554641:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.554676:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760751.554709:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758848:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760751.555231:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6061, 6061] 00000080:00200000:3.0:1789760751.555248:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.555264:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.555281:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24829952 i_blocks 49159 00000020:00200000:3.0:1789760751.555285:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.555295:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.555330:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24829952, count: 4096 00000080:00200000:3.0:1789760751.555338:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24829952, bytes: 4096 00000080:00200000:3.0:1789760751.555354:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.555377:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.555380:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24829952, 24834048) 0 0 00000080:00200000:3.0:1789760751.555386:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.555438:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6062, 6062] 00020000:00200000:3.0:1789760751.555448:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24829952 chunk: [24829952, 24834048) 24834048, 4096 00020000:00200000:3.0:1789760751.555452:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.555474:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6062, 6062] 00000008:00010000:3.0:1789760751.555492:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.555499:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.555511:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24829952, 24834048) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.555525:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24829952=0x17ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.555606:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.555610:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24834048 00000020:00010000:3.0:1789760751.555618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.555622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.555626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.555629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.555632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.555641:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.555649:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.555669:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760751.555783:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 24829952<>24834048 00000100:00100000:2.0:1789760751.557321:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760751.557356:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758976:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760751.619494:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973381 gen 3 00000008:00010000:2.0:1789760751.619559:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760751.619569:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.619579:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.619601:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760751.619631:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493758976:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760751.619719:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6062, 6062] 00000080:00200000:3.0:1789760751.619743:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.619748:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.619764:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24834048 i_blocks 49159 00000020:00200000:3.0:1789760751.619767:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.619777:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.619809:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24834048, count: 4096 00000080:00200000:3.0:1789760751.619816:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24834048, bytes: 4096 00000080:00200000:3.0:1789760751.619826:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.619835:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.619837:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24834048, 24838144) 0 0 00000080:00200000:3.0:1789760751.619843:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.619860:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6063, 6063] 00020000:00200000:3.0:1789760751.619867:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24834048 chunk: [24834048, 24838144) 24838144, 4096 00020000:00200000:3.0:1789760751.619872:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.619890:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6063, 6063] 00000008:00010000:3.0:1789760751.619907:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.619913:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.619923:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24834048, 24838144) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.619935:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24834048=0x17af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.619986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.619988:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24838144 00000020:00010000:3.0:1789760751.619996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.619999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.620002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.620005:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.620008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.620014:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.620020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.620033:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760751.620134:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 24834048<>24838144 00000100:00100000:2.0:1789760751.620287:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760751.620306:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759104:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760751.662192:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760751.662208:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760751.662220:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760751.662242:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760751.662285:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759104:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760751.662422:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6063, 6063] 00000080:00200000:3.0:1789760751.662440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.662446:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.662462:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24838144 i_blocks 49159 00000020:00200000:3.0:1789760751.662466:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.662476:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.662509:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24838144, count: 4096 00000080:00200000:3.0:1789760751.662517:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24838144, bytes: 4096 00000080:00200000:3.0:1789760751.662527:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.662536:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.662538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24838144, 24842240) 0 0 00000080:00200000:3.0:1789760751.662543:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.662558:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6064, 6064] 00020000:00200000:3.0:1789760751.662567:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24838144 chunk: [24838144, 24842240) 24842240, 4096 00020000:00200000:3.0:1789760751.662571:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.662589:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6064, 6064] 00000008:00010000:3.0:1789760751.662608:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.662615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.662625:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24838144, 24842240) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.662639:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24838144=0x17b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.662687:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.662689:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24842240 00000020:00010000:3.0:1789760751.662698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.662702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.662705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.662708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.662711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.662718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.662725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.662855:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:2.0:1789760751.662954:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 24838144<>24842240 00000100:00100000:2.0:1789760751.663007:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760751.726967:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973383 gen 3 00000008:00010000:3.0:1789760751.727027:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:3.0:1789760751.727034:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.727042:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.727059:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:3.0:1789760751.727085:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759232:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760751.727244:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6064, 6064] 00000080:00200000:2.0:1789760751.727274:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.727280:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.727290:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24842240 i_blocks 49159 00000020:00200000:2.0:1789760751.727294:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.727302:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.727327:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24842240, count: 4096 00000080:00200000:2.0:1789760751.727333:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24842240, bytes: 4096 00000080:00200000:2.0:1789760751.727341:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.727348:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.727350:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24842240, 24846336) 0 0 00000080:00200000:2.0:1789760751.727354:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.727366:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6065, 6065] 00020000:00200000:2.0:1789760751.727373:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24842240 chunk: [24842240, 24846336) 24846336, 4096 00020000:00200000:2.0:1789760751.727376:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.727390:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6065, 6065] 00000008:00010000:2.0:1789760751.727434:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.727440:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760751.727449:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24842240, 24846336) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.727460:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24842240=0x17b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.727503:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.727506:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24846336 00000020:00010000:2.0:1789760751.727512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.727515:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.727518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.727521:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.727524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.727530:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.727536:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.727550:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760751.727621:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 24842240<>24846336 00000100:00100000:3.0:1789760751.727671:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493759360:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760751.791847:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973384 gen 3 00000008:00010000:3.0:1789760751.791884:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760751.791890:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.791901:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.791922:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760751.791947:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493759360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760751.793906:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6065, 6065] 00000080:00200000:2.0:1789760751.793924:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.793930:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.793942:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24846336 i_blocks 49159 00000020:00200000:2.0:1789760751.793947:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.793959:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.793986:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24846336, count: 4096 00000080:00200000:2.0:1789760751.794002:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24846336, bytes: 4096 00000080:00200000:2.0:1789760751.794014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.794022:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.794024:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24846336, 24850432) 0 0 00000080:00200000:2.0:1789760751.794028:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.794041:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6066, 6066] 00020000:00200000:2.0:1789760751.794048:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24846336 chunk: [24846336, 24850432) 24850432, 4096 00020000:00200000:2.0:1789760751.794052:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.794068:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6066, 6066] 00000008:00010000:2.0:1789760751.794083:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.794088:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760751.794097:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24846336, 24850432) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.794109:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24846336=0x17b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.794174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.794177:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24850432 00000020:00010000:2.0:1789760751.794183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.794186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.794189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.794192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.794195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.794202:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.794208:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.794227:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760751.794565:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 24846336<>24850432 00000100:00100000:2.0:1789760751.794614:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760751.849245:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973385 gen 3 00000008:00010000:3.0:1789760751.849375:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760751.849385:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.849502:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.849564:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760751.849597:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493759488:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.853523:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759616:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760751.855118:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760751.889227:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973386 gen 3 00000008:00010000:1.0:1789760751.889287:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760751.889296:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.889307:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.889333:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760751.889362:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759616:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.890468:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760751.890489:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759744:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.923784:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973387 gen 3 00000008:00010000:1.0:1789760751.923852:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760751.923861:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.923873:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.923897:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760751.923927:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759744:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760751.924923:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759872:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760751.962756:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760751.962771:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.962784:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.962807:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760751.962837:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493759872:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760751.962923:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6069, 6069] 00000080:00200000:3.0:1789760751.962939:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.962945:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.962960:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24862720 i_blocks 49159 00000020:00200000:3.0:1789760751.962964:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.962976:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.963010:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24862720, count: 4096 00000080:00200000:3.0:1789760751.963017:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24862720, bytes: 4096 00000080:00200000:3.0:1789760751.963028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.963037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.963039:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24862720, 24866816) 0 0 00000080:00200000:3.0:1789760751.963043:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.963059:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6070, 6070] 00020000:00200000:3.0:1789760751.963068:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24862720 chunk: [24862720, 24866816) 24866816, 4096 00020000:00200000:3.0:1789760751.963072:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.963091:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6070, 6070] 00000008:00010000:3.0:1789760751.963108:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.963115:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760751.963127:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24862720, 24866816) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.963140:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24862720=0x17b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.963197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.963200:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24866816 00000020:00010000:3.0:1789760751.963209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.963213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.963216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.963219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.963222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.963229:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.963236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.963263:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760751.963379:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 24862720<>24866816 00000100:00100000:3.0:1789760751.963461:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.963479:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493760000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760751.997839:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973389 gen 3 00000008:00010000:3.0:1789760751.997908:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:3.0:1789760751.997916:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.997926:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.998009:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:3.0:1789760751.998039:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493760000:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.006018:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.006057:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760128:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.086338:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973390 gen 3 00000008:00010000:1.0:1789760752.086429:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760752.086438:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.086450:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.086473:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760752.086501:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760128:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.087275:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760256:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760752.119365:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.119382:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.119435:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.119460:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.119490:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760256:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.121563:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.121663:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760384:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760752.172930:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493760640:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760752.173200:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1789760752.173210:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493760512:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760752.173266:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493760768:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760752.177520:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973392 gen 3 00000008:00010000:1.0:1789760752.177585:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.177594:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.177603:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.177631:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.177662:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493760384:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760752.178510:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493760512:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760752.178529:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493760768:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760752.178541:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493760640:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760752.180600:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760752.180624:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493760896:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760752.268029:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760752.268044:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.268055:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.268077:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760752.268107:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493760896:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.272482:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761024:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.328618:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973394 gen 3 00000008:00010000:1.0:1789760752.328685:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760752.328694:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.328704:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.328728:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760752.328758:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761024:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.329681:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761152:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.367206:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973395 gen 3 00000008:00010000:1.0:1789760752.367271:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760752.367280:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.367291:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.367446:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760752.367477:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761152:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.370744:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761280:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.405490:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973396 gen 3 00000008:00010000:1.0:1789760752.405546:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760752.405555:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.405566:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.405591:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760752.405620:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761280:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.406572:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761408:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760752.437915:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760752.437929:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.437939:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.437959:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760752.437985:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761408:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.439046:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.439066:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761536:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.474425:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973398 gen 3 00000008:00010000:1.0:1789760752.474493:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760752.474503:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.474514:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.474538:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760752.474567:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761536:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.475488:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760752.475509:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.524719:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973399 gen 3 00000008:00010000:1.0:1789760752.524829:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.524838:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.524849:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.524925:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.524955:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.525724:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761792:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.567710:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973400 gen 3 00000008:00010000:1.0:1789760752.567774:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.567783:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.567794:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.567826:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.567858:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493761792:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.569161:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761920:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760752.597780:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760752.597797:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.597809:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.597830:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760752.597861:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493761920:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.600521:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.600559:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762048:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.639935:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973402 gen 3 00000008:00010000:1.0:1789760752.640005:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760752.640014:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.640023:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.640052:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760752.640083:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762048:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.642377:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760752.642442:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762176:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.700137:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973403 gen 3 00000008:00010000:1.0:1789760752.700192:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760752.700201:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.700212:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.700235:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760752.700276:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762176:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.701086:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762304:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.757820:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973404 gen 3 00000008:00010000:1.0:1789760752.757874:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760752.757882:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.757908:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.757934:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760752.757966:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762304:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.759123:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762432:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.808112:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973405 gen 3 00000008:00010000:1.0:1789760752.808474:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760752.808484:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.808496:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.808520:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760752.808612:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762432:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.809385:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.809430:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762560:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760752.862113:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973406 gen 3 00000008:00010000:1.0:1789760752.862171:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760752.862180:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.862192:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.862213:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760752.862242:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762560:192.168.203.160@tcp:4:dd.0 00000020:00010000:0.0:1789760752.862914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.862918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.862921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.862928:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.862935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760752.862951:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760752.863052:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 24936448<>24940544 00000100:00100000:1.0:1789760752.863205:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760752.863224:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762688:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760752.878001:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760752.916258:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973407 gen 3 00000008:00010000:1.0:1789760752.916316:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.916324:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.916336:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.916359:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.916387:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493762688:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760752.916620:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6088, 6088] 00000080:00200000:0.0:1789760752.916636:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760752.916642:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760752.916659:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24940544 i_blocks 49159 00000020:00200000:0.0:1789760752.916663:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760752.916674:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760752.916709:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24940544, count: 4096 00000080:00200000:0.0:1789760752.916716:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24940544, bytes: 4096 00000080:00200000:0.0:1789760752.916725:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760752.916734:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760752.916737:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24940544, 24944640) 0 0 00000080:00200000:0.0:1789760752.916741:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760752.916758:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6089, 6089] 00020000:00200000:0.0:1789760752.916766:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24940544 chunk: [24940544, 24944640) 24944640, 4096 00020000:00200000:0.0:1789760752.916771:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760752.916790:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6089, 6089] 00000008:00010000:0.0:1789760752.916808:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760752.916814:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760752.916826:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24940544, 24944640) append 0 append lockpos 0 00000080:00200000:0.0:1789760752.916839:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24940544=0x17c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760752.916896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760752.916898:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24944640 00000020:00010000:0.0:1789760752.916906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:0.0:1789760752.916910:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.916914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.916917:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.916920:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.916927:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.916934:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760752.916949:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760752.917056:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 24940544<>24944640 00000100:00100000:1.0:1789760752.917212:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.917230:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762816:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760752.965908:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760752.965923:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.965935:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.966007:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760752.966049:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762816:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760752.968848:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6089, 6089] 00000080:00200000:0.0:1789760752.968877:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760752.968884:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760752.968902:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24944640 i_blocks 49159 00000020:00200000:0.0:1789760752.968908:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760752.968924:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760752.968959:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24944640, count: 4096 00000080:00200000:0.0:1789760752.968968:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24944640, bytes: 4096 00000080:00200000:0.0:1789760752.968983:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760752.968993:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760752.968995:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24944640, 24948736) 0 0 00000080:00200000:0.0:1789760752.969000:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760752.969016:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6090, 6090] 00020000:00200000:0.0:1789760752.969025:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24944640 chunk: [24944640, 24948736) 24948736, 4096 00020000:00200000:0.0:1789760752.969029:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760752.971018:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6090, 6090] 00000008:00010000:0.0:1789760752.971064:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760752.971071:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760752.971086:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24944640, 24948736) append 0 append lockpos 0 00000080:00200000:0.0:1789760752.971105:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24944640=0x17ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760752.971190:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760752.971194:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24948736 00000020:00010000:0.0:1789760752.971203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:0.0:1789760752.971207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.971221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760752.971226:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760752.971229:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760752.971242:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760752.971259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760752.971287:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760752.971667:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 24944640<>24948736 00000100:00100000:1.0:1789760752.972128:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762944:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760753.053603:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973409 gen 3 00000008:00010000:1.0:1789760753.053672:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760753.053681:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.053691:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.053720:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760753.053747:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493762944:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760753.054543:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6090, 6090] 00000080:00200000:0.0:1789760753.054564:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.054571:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.054590:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24948736 i_blocks 49159 00000020:00200000:0.0:1789760753.054777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.054791:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.054825:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24948736, count: 4096 00000080:00200000:0.0:1789760753.054832:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24948736, bytes: 4096 00000080:00200000:0.0:1789760753.054847:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.054856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.054858:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24948736, 24952832) 0 0 00000080:00200000:0.0:1789760753.054863:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.054888:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6091, 6091] 00020000:00200000:0.0:1789760753.054897:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24948736 chunk: [24948736, 24952832) 24952832, 4096 00020000:00200000:0.0:1789760753.054901:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.054920:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6091, 6091] 00000008:00010000:0.0:1789760753.054937:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.054943:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760753.054954:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24948736, 24952832) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.055182:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24948736=0x17cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.055254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.055258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24952832 00000020:00010000:0.0:1789760753.055266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.055269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.055274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.055277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.055280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.055288:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.055294:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.055313:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760753.056024:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 24948736<>24952832 00000100:00100000:1.0:1789760753.056209:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493763072:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760753.099252:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973410 gen 3 00000008:00010000:1.0:1789760753.099309:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760753.099319:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.099329:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.099435:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760753.099463:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493763072:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760753.099682:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6091, 6091] 00000080:00200000:0.0:1789760753.099697:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760753.099704:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760753.099738:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24952832 i_blocks 49159 00000020:00200000:0.0:1789760753.099742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760753.099752:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760753.099785:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24952832, count: 4096 00000080:00200000:0.0:1789760753.099792:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24952832, bytes: 4096 00000080:00200000:0.0:1789760753.099801:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760753.099810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760753.099812:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24952832, 24956928) 0 0 00000080:00200000:0.0:1789760753.099816:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760753.099831:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6092, 6092] 00020000:00200000:0.0:1789760753.099838:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24952832 chunk: [24952832, 24956928) 24956928, 4096 00020000:00200000:0.0:1789760753.099843:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760753.099860:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6092, 6092] 00000008:00010000:0.0:1789760753.099876:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760753.099882:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760753.099892:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24952832, 24956928) append 0 append lockpos 0 00000080:00200000:0.0:1789760753.099905:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24952832=0x17cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760753.099955:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760753.099957:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24956928 00000020:00010000:0.0:1789760753.099964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:0.0:1789760753.099968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.099972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760753.099976:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760753.099978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760753.099985:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760753.099992:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760753.100003:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760753.100101:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 24952832<>24956928 00000100:00100000:1.0:1789760753.100167:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493763200:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760753.127113:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973411 gen 3 00000008:00010000:1.0:1789760753.127167:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760753.127177:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.127189:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.127275:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760753.127307:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493763200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760753.130852:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6092, 6092] 00000080:00200000:2.0:1789760753.130882:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.130889:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.130918:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24956928 i_blocks 49159 00000020:00200000:2.0:1789760753.130923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.130940:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.130984:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24956928, count: 4096 00000080:00200000:2.0:1789760753.130993:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24956928, bytes: 4096 00000080:00200000:2.0:1789760753.131009:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.131020:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.131022:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24956928, 24961024) 0 0 00000080:00200000:2.0:1789760753.131027:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.131043:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6093, 6093] 00020000:00200000:2.0:1789760753.131053:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24956928 chunk: [24956928, 24961024) 24961024, 4096 00020000:00200000:2.0:1789760753.131057:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.131077:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6093, 6093] 00000008:00010000:2.0:1789760753.131096:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.131102:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760753.131114:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24956928, 24961024) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.131128:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24956928=0x17cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.131209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.131212:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24961024 00000020:00010000:2.0:1789760753.131222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.131225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.131229:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.131232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.131235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.131261:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.131268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.131294:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760753.131667:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 24956928<>24961024 00000100:00100000:2.0:1789760753.131728:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493763328:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760753.188818:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760753.188835:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.188848:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.188867:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760753.188898:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493763328:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.188990:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6093, 6093] 00000080:00200000:3.0:1789760753.189005:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.189011:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.189027:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24961024 i_blocks 49159 00000020:00200000:3.0:1789760753.189031:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.189042:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.189075:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24961024, count: 4096 00000080:00200000:3.0:1789760753.189083:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24961024, bytes: 4096 00000080:00200000:3.0:1789760753.189093:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.189101:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.189103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24961024, 24965120) 0 0 00000080:00200000:3.0:1789760753.189109:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.189124:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6094, 6094] 00020000:00200000:3.0:1789760753.189132:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24961024 chunk: [24961024, 24965120) 24965120, 4096 00020000:00200000:3.0:1789760753.189137:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.189156:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6094, 6094] 00000008:00010000:3.0:1789760753.189173:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.189178:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.189190:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24961024, 24965120) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.189204:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24961024=0x17ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.189274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.189277:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24965120 00000020:00010000:3.0:1789760753.189285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.189289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.189292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.189295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.189298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.189306:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.189313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.189328:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760753.189689:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 24961024<>24965120 00000100:00100000:3.0:1789760753.189733:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763456:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.231921:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973413 gen 3 00000008:00010000:2.0:1789760753.231990:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760753.231998:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.232008:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.232023:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760753.232052:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763456:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.232314:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6094, 6094] 00000080:00200000:3.0:1789760753.232330:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.232335:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.232350:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24965120 i_blocks 49159 00000020:00200000:3.0:1789760753.232354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.232363:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.232417:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24965120, count: 4096 00000080:00200000:3.0:1789760753.232426:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24965120, bytes: 4096 00000080:00200000:3.0:1789760753.232436:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.232447:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.232449:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24965120, 24969216) 0 0 00000080:00200000:3.0:1789760753.232454:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.232516:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6095, 6095] 00020000:00200000:3.0:1789760753.232526:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24965120 chunk: [24965120, 24969216) 24969216, 4096 00020000:00200000:3.0:1789760753.232530:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.232550:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6095, 6095] 00000008:00010000:3.0:1789760753.232566:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.232571:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.232582:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24965120, 24969216) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.232593:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24965120=0x17cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.232643:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.232645:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24969216 00000020:00010000:3.0:1789760753.232653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.232656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.232658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.232661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.232663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.232669:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.232674:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.232687:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760753.232790:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 24965120<>24969216 00000100:00100000:2.0:1789760753.233008:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760753.233030:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763584:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760753.269807:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760753.269926:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.269937:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.269994:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760753.270025:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763584:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.271937:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6095, 6095] 00000080:00200000:3.0:1789760753.271964:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.271971:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.271990:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24969216 i_blocks 49159 00000020:00200000:3.0:1789760753.271997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.272014:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.272052:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24969216, count: 4096 00000080:00200000:3.0:1789760753.272061:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24969216, bytes: 4096 00000080:00200000:3.0:1789760753.272183:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.272194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.272196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24969216, 24973312) 0 0 00000080:00200000:3.0:1789760753.272201:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.272219:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6096, 6096] 00020000:00200000:3.0:1789760753.272229:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24969216 chunk: [24969216, 24973312) 24973312, 4096 00020000:00200000:3.0:1789760753.272234:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.272269:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6096, 6096] 00000008:00010000:3.0:1789760753.272289:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.272295:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.272307:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24969216, 24973312) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.272960:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24969216=0x17d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.273056:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.273059:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24973312 00000020:00010000:3.0:1789760753.273071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.273075:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.273078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.273083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.273086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.273098:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.273106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.273135:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760753.273269:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 24969216<>24973312 00000100:00100000:2.0:1789760753.273382:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763712:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.304715:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973415 gen 3 00000008:00010000:2.0:1789760753.304790:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760753.304798:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.304808:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.304833:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760753.304861:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763712:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.305060:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6096, 6096] 00000080:00200000:3.0:1789760753.305076:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.305083:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.305099:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24973312 i_blocks 49159 00000020:00200000:3.0:1789760753.305103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.305113:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.305147:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24973312, count: 4096 00000080:00200000:3.0:1789760753.305155:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24973312, bytes: 4096 00000080:00200000:3.0:1789760753.305165:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.305175:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.305177:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24973312, 24977408) 0 0 00000080:00200000:3.0:1789760753.305182:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.305198:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6097, 6097] 00020000:00200000:3.0:1789760753.305206:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24973312 chunk: [24973312, 24977408) 24977408, 4096 00020000:00200000:3.0:1789760753.305213:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.305231:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6097, 6097] 00000008:00010000:3.0:1789760753.305261:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.305267:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.305279:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24973312, 24977408) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.305303:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24973312=0x17d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.305362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.305364:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24977408 00000020:00010000:3.0:1789760753.305373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.305376:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.305379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.305383:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.305385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.305432:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.305440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.305456:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760753.305560:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 24973312<>24977408 00000100:00100000:2.0:1789760753.305802:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760753.305831:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763840:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760753.342036:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760753.342052:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.342063:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.342083:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760753.342117:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763840:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.342387:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6097, 6097] 00000080:00200000:3.0:1789760753.342431:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.342438:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.342453:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24977408 i_blocks 49159 00000020:00200000:3.0:1789760753.342457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.342467:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.342502:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24977408, count: 4096 00000080:00200000:3.0:1789760753.342510:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24977408, bytes: 4096 00000080:00200000:3.0:1789760753.342521:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.342531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.342533:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24977408, 24981504) 0 0 00000080:00200000:3.0:1789760753.342539:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.342555:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6098, 6098] 00020000:00200000:3.0:1789760753.342566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24977408 chunk: [24977408, 24981504) 24981504, 4096 00020000:00200000:3.0:1789760753.342571:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.342591:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6098, 6098] 00000008:00010000:3.0:1789760753.342609:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.342615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.342627:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24977408, 24981504) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.342641:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24977408=0x17d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.342695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.342698:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24981504 00000020:00010000:3.0:1789760753.342706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.342710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.342713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.342717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.342720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.342727:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.342734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.342750:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760753.342924:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 24977408<>24981504 00000100:00100000:2.0:1789760753.342989:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763968:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.383889:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973416 gen 3 00000100:00100000:2.0:1789760753.384188:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149180 x1876699493763968/t4294973417(4294973417) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760769 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760753.384250:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760753.384258:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.384268:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.384346:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760753.384382:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493763968:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.384631:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6098, 6098] 00000080:00200000:3.0:1789760753.384649:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.384656:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.384672:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24981504 i_blocks 49159 00000020:00200000:3.0:1789760753.384675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.384684:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.384723:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24981504, count: 4096 00000080:00200000:3.0:1789760753.384731:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24981504, bytes: 4096 00000080:00200000:3.0:1789760753.384744:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.384755:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.384757:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24981504, 24985600) 0 0 00000080:00200000:3.0:1789760753.384762:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.384777:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6099, 6099] 00020000:00200000:3.0:1789760753.384787:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24981504 chunk: [24981504, 24985600) 24985600, 4096 00020000:00200000:3.0:1789760753.384791:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.385176:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6099, 6099] 00000008:00010000:3.0:1789760753.385196:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.385202:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.385215:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24981504, 24985600) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.385230:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24981504=0x17d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.385367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.385370:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24985600 00000020:00010000:3.0:1789760753.385380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.385384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.385387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.385390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.385411:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.385420:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.385427:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.385444:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760753.385550:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 24981504<>24985600 00000100:00100000:2.0:1789760753.385619:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764096:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.422464:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973418 gen 3 00000008:00010000:2.0:1789760753.422534:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760753.422886:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.422899:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.422920:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760753.422952:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764096:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.423256:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6099, 6099] 00000080:00200000:3.0:1789760753.423272:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.423278:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.423356:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24985600 i_blocks 49159 00000020:00200000:3.0:1789760753.423361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.423371:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.423443:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24985600, count: 4096 00000080:00200000:3.0:1789760753.423452:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24985600, bytes: 4096 00000080:00200000:3.0:1789760753.423463:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.423472:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.423474:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24985600, 24989696) 0 0 00000080:00200000:3.0:1789760753.423480:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.423497:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6100, 6100] 00020000:00200000:3.0:1789760753.423505:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24985600 chunk: [24985600, 24989696) 24989696, 4096 00020000:00200000:3.0:1789760753.423509:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.423529:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6100, 6100] 00000008:00010000:3.0:1789760753.423556:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.423683:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.423698:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24985600, 24989696) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.423712:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24985600=0x17d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.423959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.423962:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24989696 00000020:00010000:3.0:1789760753.423971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.423975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.423977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.423980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.423983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.423992:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.423999:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.424017:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760753.424188:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 24985600<>24989696 00000100:00100000:2.0:1789760753.424495:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764224:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.492355:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973419 gen 3 00000008:00010000:2.0:1789760753.492452:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760753.492461:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.492473:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.492493:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760753.492522:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764224:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.492941:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6100, 6100] 00000080:00200000:3.0:1789760753.492956:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.492962:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.492976:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24989696 i_blocks 49159 00000020:00200000:3.0:1789760753.492980:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.492989:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.493022:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24989696, count: 4096 00000080:00200000:3.0:1789760753.493028:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24989696, bytes: 4096 00000080:00200000:3.0:1789760753.493038:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.493047:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.493050:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24989696, 24993792) 0 0 00000080:00200000:3.0:1789760753.493054:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.493068:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6101, 6101] 00020000:00200000:3.0:1789760753.493077:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24989696 chunk: [24989696, 24993792) 24993792, 4096 00020000:00200000:3.0:1789760753.493081:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.493100:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6101, 6101] 00000008:00010000:3.0:1789760753.493116:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.493122:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.493133:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24989696, 24993792) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.493146:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24989696=0x17d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.493198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.493200:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24993792 00000020:00010000:3.0:1789760753.493208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.493211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.493214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.493218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.493221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.493228:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.493235:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.493330:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760753.494333:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 24989696<>24993792 00000100:00100000:2.0:1789760753.494510:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764352:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.557588:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973420 gen 3 00000008:00010000:2.0:1789760753.557638:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760753.557646:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.557657:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.557681:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760753.557710:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764352:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.557832:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6101, 6101] 00000080:00200000:3.0:1789760753.557847:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.557853:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.557867:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24993792 i_blocks 49159 00000020:00200000:3.0:1789760753.557871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.557880:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.557922:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24993792, count: 4096 00000080:00200000:3.0:1789760753.557929:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24993792, bytes: 4096 00000080:00200000:3.0:1789760753.557944:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.557954:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.557957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24993792, 24997888) 0 0 00000080:00200000:3.0:1789760753.557962:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.557977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6102, 6102] 00020000:00200000:3.0:1789760753.557986:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24993792 chunk: [24993792, 24997888) 24997888, 4096 00020000:00200000:3.0:1789760753.557990:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.558010:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6102, 6102] 00000008:00010000:3.0:1789760753.558027:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.558032:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.558044:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24993792, 24997888) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.558058:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24993792=0x17d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.558112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.558115:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24997888 00000020:00010000:3.0:1789760753.558123:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.558127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.558130:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.558133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.558136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.558143:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.558150:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.558165:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760753.558277:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 24993792<>24997888 00000100:00100000:2.0:1789760753.560199:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764480:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.606791:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973421 gen 3 00000008:00010000:2.0:1789760753.606837:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760753.606845:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.606856:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.606916:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760753.606945:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764480:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.607042:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6102, 6102] 00000080:00200000:3.0:1789760753.607058:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.607064:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.607078:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24997888 i_blocks 49159 00000020:00200000:3.0:1789760753.607082:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.607091:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.607121:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24997888, count: 4096 00000080:00200000:3.0:1789760753.607127:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24997888, bytes: 4096 00000080:00200000:3.0:1789760753.607136:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.607145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.607147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24997888, 25001984) 0 0 00000080:00200000:3.0:1789760753.607152:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.607167:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6103, 6103] 00020000:00200000:3.0:1789760753.607175:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24997888 chunk: [24997888, 25001984) 25001984, 4096 00020000:00200000:3.0:1789760753.607179:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.607197:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6103, 6103] 00000008:00010000:3.0:1789760753.607214:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.607220:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.607231:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24997888, 25001984) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.607254:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=24997888=0x17d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.608873:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.608882:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25001984 00000020:00010000:3.0:1789760753.608897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.608901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.608905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.608910:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.608912:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.608925:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.608933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.608957:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760753.609062:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 24997888<>25001984 00000100:00100000:2.0:1789760753.609845:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764608:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.646238:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973422 gen 3 00000008:00010000:2.0:1789760753.646310:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760753.646320:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.646330:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.646352:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760753.646384:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764608:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.646559:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6103, 6103] 00000080:00200000:3.0:1789760753.646578:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.646586:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.646603:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25001984 i_blocks 49159 00000020:00200000:3.0:1789760753.646607:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.646618:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.646655:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25001984, count: 4096 00000080:00200000:3.0:1789760753.646664:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25001984, bytes: 4096 00000080:00200000:3.0:1789760753.646673:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.646683:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.646686:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25001984, 25006080) 0 0 00000080:00200000:3.0:1789760753.646691:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.646706:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6104, 6104] 00020000:00200000:3.0:1789760753.646715:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25001984 chunk: [25001984, 25006080) 25006080, 4096 00020000:00200000:3.0:1789760753.646730:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.646751:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6104, 6104] 00000008:00010000:3.0:1789760753.646771:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.646777:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.646789:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25001984, 25006080) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.646804:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25001984=0x17d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.646864:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.646867:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25006080 00000020:00010000:3.0:1789760753.646876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.646880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.646883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.646887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.646891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.646899:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.646906:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.646924:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760753.647032:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25001984<>25006080 00000100:00100000:2.0:1789760753.647208:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764736:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760753.687041:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760753.687057:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.687069:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.687092:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760753.687124:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764736:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.687323:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6104, 6104] 00000080:00200000:3.0:1789760753.687339:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.687345:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.687360:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25006080 i_blocks 49159 00000020:00200000:3.0:1789760753.687364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.687373:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.687432:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25006080, count: 4096 00000080:00200000:3.0:1789760753.687439:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25006080, bytes: 4096 00000080:00200000:3.0:1789760753.687448:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.687458:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.687460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25006080, 25010176) 0 0 00000080:00200000:3.0:1789760753.687465:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.687479:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6105, 6105] 00020000:00200000:3.0:1789760753.687488:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25006080 chunk: [25006080, 25010176) 25010176, 4096 00020000:00200000:3.0:1789760753.687492:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.687510:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6105, 6105] 00000008:00010000:3.0:1789760753.687526:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.687532:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.687543:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25006080, 25010176) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.687557:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25006080=0x17d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.687606:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.687609:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25010176 00000020:00010000:3.0:1789760753.687616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.687620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.687623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.687626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.687629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.687805:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.687813:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.687878:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760753.687983:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25006080<>25010176 00000100:00100000:2.0:1789760753.688092:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764864:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.727151:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973424 gen 3 00000008:00010000:2.0:1789760753.727221:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760753.727230:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.727251:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.727271:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760753.727309:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493764864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.727477:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6105, 6105] 00000080:00200000:3.0:1789760753.727494:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.727500:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.727516:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25010176 i_blocks 49159 00000020:00200000:3.0:1789760753.727520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.727530:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.727567:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25010176, count: 4096 00000080:00200000:3.0:1789760753.727576:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25010176, bytes: 4096 00000080:00200000:3.0:1789760753.727588:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.727598:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.727600:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25010176, 25014272) 0 0 00000080:00200000:3.0:1789760753.727605:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.727626:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6106, 6106] 00020000:00200000:3.0:1789760753.727645:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25010176 chunk: [25010176, 25014272) 25014272, 4096 00020000:00200000:3.0:1789760753.727650:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.727675:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6106, 6106] 00000008:00010000:3.0:1789760753.727698:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.727705:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.727727:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25010176, 25014272) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.727747:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25010176=0x17da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.727810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.727813:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25014272 00000020:00010000:3.0:1789760753.727822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.727825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.727829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.727833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.727836:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.727844:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.727850:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.727867:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760753.727970:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25010176<>25014272 00000100:00100000:2.0:1789760753.728180:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764992:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.769717:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973425 gen 3 00000008:00010000:2.0:1789760753.769780:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760753.769790:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.769800:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.769830:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760753.769859:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493764992:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.769958:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6106, 6106] 00000080:00200000:3.0:1789760753.769976:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.769982:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.769998:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25014272 i_blocks 49159 00000020:00200000:3.0:1789760753.770002:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.770012:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.770047:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25014272, count: 4096 00000080:00200000:3.0:1789760753.770055:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25014272, bytes: 4096 00000080:00200000:3.0:1789760753.770064:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.770073:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.770076:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25014272, 25018368) 0 0 00000080:00200000:3.0:1789760753.770081:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.770100:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6107, 6107] 00020000:00200000:3.0:1789760753.770109:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25014272 chunk: [25014272, 25018368) 25018368, 4096 00020000:00200000:3.0:1789760753.770114:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.770134:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6107, 6107] 00000008:00010000:3.0:1789760753.770152:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.770159:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.770170:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25014272, 25018368) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.770183:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25014272=0x17db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.770253:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.770257:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25018368 00000020:00010000:3.0:1789760753.770266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.770270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.770273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.770277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.770280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.770288:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.770296:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.770311:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760753.770452:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25014272<>25018368 00000100:00100000:2.0:1789760753.771144:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765120:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.833137:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973426 gen 3 00000008:00010000:2.0:1789760753.833193:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760753.833201:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.833214:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.833252:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760753.833281:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.833387:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6107, 6107] 00000080:00200000:3.0:1789760753.833429:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.833435:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.833452:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25018368 i_blocks 49159 00000020:00200000:3.0:1789760753.833457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.833467:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.833500:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25018368, count: 4096 00000080:00200000:3.0:1789760753.833508:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25018368, bytes: 4096 00000080:00200000:3.0:1789760753.833518:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.833527:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.833529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25018368, 25022464) 0 0 00000080:00200000:3.0:1789760753.833534:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.833551:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6108, 6108] 00020000:00200000:3.0:1789760753.833559:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25018368 chunk: [25018368, 25022464) 25022464, 4096 00020000:00200000:3.0:1789760753.833564:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.833583:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6108, 6108] 00000008:00010000:3.0:1789760753.833600:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.833606:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.833618:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25018368, 25022464) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.833631:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25018368=0x17dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.833685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.833688:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25022464 00000020:00010000:3.0:1789760753.833696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.833699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.833703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.833706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.833709:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.833726:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.833733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.833749:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760753.834122:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25018368<>25022464 00000100:00100000:2.0:1789760753.834190:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765248:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760753.872599:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760753.872616:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.872627:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.872691:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760753.872731:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765248:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.873353:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6108, 6108] 00000080:00200000:3.0:1789760753.873370:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.873377:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.873426:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25022464 i_blocks 49159 00000020:00200000:3.0:1789760753.873432:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.873442:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.873478:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25022464, count: 4096 00000080:00200000:3.0:1789760753.873486:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25022464, bytes: 4096 00000080:00200000:3.0:1789760753.873500:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.873511:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.873513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25022464, 25026560) 0 0 00000080:00200000:3.0:1789760753.873518:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.873535:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6109, 6109] 00020000:00200000:3.0:1789760753.873543:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25022464 chunk: [25022464, 25026560) 25026560, 4096 00020000:00200000:3.0:1789760753.873547:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.873567:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6109, 6109] 00000008:00010000:3.0:1789760753.873587:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.873593:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.873605:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25022464, 25026560) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.873619:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25022464=0x17dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.873676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.873680:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25026560 00000020:00010000:3.0:1789760753.873688:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.873692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.873696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.873700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.873702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.873710:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.873778:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.873799:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760753.873964:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25022464<>25026560 00000100:00100000:2.0:1789760753.874094:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765376:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760753.901699:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760753.913715:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973428 gen 3 00000008:00010000:2.0:1789760753.913785:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760753.913794:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.913814:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.913833:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760753.913871:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765376:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.913929:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6109, 6109] 00000080:00200000:3.0:1789760753.913945:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.913950:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.913964:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25026560 i_blocks 49159 00000020:00200000:3.0:1789760753.913968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.913977:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.914866:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25026560, count: 4096 00000080:00200000:3.0:1789760753.914881:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25026560, bytes: 4096 00000080:00200000:3.0:1789760753.914897:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.914906:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.914908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25026560, 25030656) 0 0 00000080:00200000:3.0:1789760753.914915:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.914931:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6110, 6110] 00020000:00200000:3.0:1789760753.914941:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25026560 chunk: [25026560, 25030656) 25030656, 4096 00020000:00200000:3.0:1789760753.914945:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.914965:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6110, 6110] 00000008:00010000:3.0:1789760753.914983:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.914990:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.915003:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25026560, 25030656) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.915018:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25026560=0x17de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.915083:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.915086:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25030656 00000020:00010000:3.0:1789760753.915094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.915098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.915101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.915105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.915108:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.915119:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.915126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.915148:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760753.915264:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25026560<>25030656 00000100:00100000:2.0:1789760753.916527:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765504:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760753.977698:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973429 gen 3 00000008:00010000:2.0:1789760753.977741:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760753.977748:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760753.977768:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760753.977793:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760753.977822:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760753.978232:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6110, 6110] 00000080:00200000:3.0:1789760753.978258:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760753.978265:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760753.978284:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25030656 i_blocks 49159 00000020:00200000:3.0:1789760753.978288:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760753.978299:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760753.978338:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25030656, count: 4096 00000080:00200000:3.0:1789760753.978345:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25030656, bytes: 4096 00000080:00200000:3.0:1789760753.978355:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760753.978365:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760753.978367:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25030656, 25034752) 0 0 00000080:00200000:3.0:1789760753.978372:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760753.978385:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6111, 6111] 00020000:00200000:3.0:1789760753.978420:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25030656 chunk: [25030656, 25034752) 25034752, 4096 00020000:00200000:3.0:1789760753.978425:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760753.978444:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6111, 6111] 00000008:00010000:3.0:1789760753.978465:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760753.978471:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760753.978484:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25030656, 25034752) append 0 append lockpos 0 00000080:00200000:3.0:1789760753.978498:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25030656=0x17df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760753.978560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760753.978564:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25034752 00000020:00010000:3.0:1789760753.978571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760753.978574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.978578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760753.978581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760753.978583:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760753.978592:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760753.978599:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760753.978617:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760753.978710:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25030656<>25034752 00000100:00100000:2.0:1789760753.978790:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765632:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.023586:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973430 gen 3 00000008:00010000:2.0:1789760754.023644:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760754.023653:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.023722:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.023788:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760754.023828:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.024194:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6111, 6111] 00000080:00200000:3.0:1789760754.024212:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.024218:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.024234:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25034752 i_blocks 49159 00000020:00200000:3.0:1789760754.024248:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.024259:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.024295:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25034752, count: 4096 00000080:00200000:3.0:1789760754.024302:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25034752, bytes: 4096 00000080:00200000:3.0:1789760754.024315:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.024325:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.024327:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25034752, 25038848) 0 0 00000080:00200000:3.0:1789760754.024332:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.024488:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6112, 6112] 00020000:00200000:3.0:1789760754.024498:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25034752 chunk: [25034752, 25038848) 25038848, 4096 00020000:00200000:3.0:1789760754.024503:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.024523:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6112, 6112] 00000008:00010000:3.0:1789760754.024542:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.024549:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.024563:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25034752, 25038848) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.024582:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25034752=0x17e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.024643:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.024646:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25038848 00000020:00010000:3.0:1789760754.024655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.024658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.024661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.024665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.024668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.024677:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.024683:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.024703:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760754.024808:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25034752<>25038848 00000100:00100000:2.0:1789760754.024948:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765760:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760754.070656:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760754.070673:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.070685:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.070708:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760754.070743:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493765760:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.071023:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6112, 6112] 00000080:00200000:3.0:1789760754.071040:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.071046:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.071061:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25038848 i_blocks 49159 00000020:00200000:3.0:1789760754.071065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.071075:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.071110:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25038848, count: 4096 00000080:00200000:3.0:1789760754.071116:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25038848, bytes: 4096 00000080:00200000:3.0:1789760754.071125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.071134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.071136:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25038848, 25042944) 0 0 00000080:00200000:3.0:1789760754.071141:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.071157:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6113, 6113] 00020000:00200000:3.0:1789760754.071167:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25038848 chunk: [25038848, 25042944) 25042944, 4096 00020000:00200000:3.0:1789760754.071171:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.071190:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6113, 6113] 00000008:00010000:3.0:1789760754.071659:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.071664:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.071677:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25038848, 25042944) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.071692:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25038848=0x17e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.071757:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.071760:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25042944 00000020:00010000:3.0:1789760754.071769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.071773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.071776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.071780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.071783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.071791:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.071798:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.071819:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760754.071927:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25038848<>25042944 00000100:00100000:2.0:1789760754.072556:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765888:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.123203:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973432 gen 3 00000008:00010000:2.0:1789760754.123287:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760754.123296:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.123308:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.123338:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760754.123371:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493765888:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.123572:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6113, 6113] 00000080:00200000:3.0:1789760754.123588:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.123595:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.123610:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25042944 i_blocks 49159 00000020:00200000:3.0:1789760754.123614:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.123623:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.123659:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25042944, count: 4096 00000080:00200000:3.0:1789760754.123668:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25042944, bytes: 4096 00000080:00200000:3.0:1789760754.123682:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.123691:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.123694:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25042944, 25047040) 0 0 00000080:00200000:3.0:1789760754.123699:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.123715:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6114, 6114] 00020000:00200000:3.0:1789760754.123724:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25042944 chunk: [25042944, 25047040) 25047040, 4096 00020000:00200000:3.0:1789760754.123729:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.123751:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6114, 6114] 00000008:00010000:3.0:1789760754.123770:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.123776:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.123789:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25042944, 25047040) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.123804:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25042944=0x17e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.123861:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.123864:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25047040 00000020:00010000:3.0:1789760754.123874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.123879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.123882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.123885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.123888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.123896:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.123902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.123989:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:2.0:1789760754.124117:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25042944<>25047040 00000100:00100000:2.0:1789760754.124157:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760754.124176:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493766016:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760754.176001:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760754.176017:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.176031:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.176060:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760754.176476:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493766016:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.177298:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6114, 6114] 00000080:00200000:3.0:1789760754.177316:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.177322:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.177338:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25047040 i_blocks 49159 00000020:00200000:3.0:1789760754.177344:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.177356:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.177391:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25047040, count: 4096 00000080:00200000:3.0:1789760754.177497:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25047040, bytes: 4096 00000080:00200000:3.0:1789760754.177513:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.177522:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.177525:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25047040, 25051136) 0 0 00000080:00200000:3.0:1789760754.177530:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.177546:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6115, 6115] 00020000:00200000:3.0:1789760754.177555:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25047040 chunk: [25047040, 25051136) 25051136, 4096 00020000:00200000:3.0:1789760754.177560:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.177581:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6115, 6115] 00000008:00010000:3.0:1789760754.177598:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.177605:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.177616:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25047040, 25051136) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.177631:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25047040=0x17e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.177690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.177694:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25051136 00000020:00010000:3.0:1789760754.177702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.177705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.177708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.177712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.178266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.178281:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.178290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.178312:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760754.178453:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25047040<>25051136 00000100:00100000:3.0:1789760754.178508:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760754.178534:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766144:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.212207:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973434 gen 3 00000008:00010000:2.0:1789760754.212290:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760754.212309:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.212319:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.212341:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760754.212372:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766144:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.212512:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6115, 6115] 00000080:00200000:3.0:1789760754.212530:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.212537:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.212555:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25051136 i_blocks 49159 00000020:00200000:3.0:1789760754.212559:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.212569:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.212602:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25051136, count: 4096 00000080:00200000:3.0:1789760754.212611:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25051136, bytes: 4096 00000080:00200000:3.0:1789760754.212622:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.212632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.212635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25051136, 25055232) 0 0 00000080:00200000:3.0:1789760754.212640:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.212657:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6116, 6116] 00020000:00200000:3.0:1789760754.212665:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25051136 chunk: [25051136, 25055232) 25055232, 4096 00020000:00200000:3.0:1789760754.212669:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.212688:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6116, 6116] 00000008:00010000:3.0:1789760754.212707:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.212713:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.212735:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25051136, 25055232) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.212750:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25051136=0x17e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.212801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.212804:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25055232 00000020:00010000:3.0:1789760754.212811:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.212815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.212818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.212822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.212825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.212832:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.212841:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.212906:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:2.0:1789760754.213019:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25051136<>25055232 00000100:00100000:2.0:1789760754.213070:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766272:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.251345:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973435 gen 3 00000008:00010000:2.0:1789760754.251425:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760754.251435:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.251447:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.251468:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760754.251497:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766272:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.251536:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6116, 6116] 00000080:00200000:3.0:1789760754.251553:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.251559:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.251577:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25055232 i_blocks 49159 00000020:00200000:3.0:1789760754.251581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.251669:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.251707:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25055232, count: 4096 00000080:00200000:2.0:1789760754.251787:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25055232, bytes: 4096 00000080:00200000:2.0:1789760754.251800:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.251810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.251813:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25055232, 25059328) 0 0 00000080:00200000:2.0:1789760754.251818:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.251837:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6117, 6117] 00020000:00200000:2.0:1789760754.251847:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25055232 chunk: [25055232, 25059328) 25059328, 4096 00020000:00200000:2.0:1789760754.251852:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.251874:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6117, 6117] 00000008:00010000:2.0:1789760754.251894:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.251901:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760754.251914:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25055232, 25059328) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.251929:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25055232=0x17e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.251991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.251993:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25059328 00000020:00010000:2.0:1789760754.252003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.252007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.252010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.252013:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacf38 (ffff8eb6450b4e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.252016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.252024:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.252031:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.252079:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760754.252166:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25055232<>25059328 00000100:00100000:2.0:1789760754.252198:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760754.252210:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766400:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.303526:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973436 gen 3 00000008:00010000:2.0:1789760754.303590:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760754.303601:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.303613:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.303635:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760754.303666:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493766400:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.303814:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6117, 6117] 00000080:00200000:3.0:1789760754.303892:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.303898:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.303915:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25059328 i_blocks 49159 00000020:00200000:3.0:1789760754.303919:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.303932:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.303999:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25059328, count: 4096 00000080:00200000:3.0:1789760754.304008:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25059328, bytes: 4096 00000080:00200000:3.0:1789760754.304078:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.304089:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.304091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25059328, 25063424) 0 0 00000080:00200000:3.0:1789760754.304096:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.304114:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6118, 6118] 00020000:00200000:3.0:1789760754.304123:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25059328 chunk: [25059328, 25063424) 25063424, 4096 00020000:00200000:3.0:1789760754.304128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.304149:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6118, 6118] 00000008:00010000:3.0:1789760754.304167:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.304174:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.304186:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25059328, 25063424) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.304199:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25059328=0x17e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.304272:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.304275:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25063424 00000020:00010000:3.0:1789760754.304283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.304287:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.304290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.304293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.304296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.304305:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.304312:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.304676:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760754.305011:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25059328<>25063424 00000100:00100000:3.0:1789760754.305136:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760754.305162:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493766528:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760754.337842:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760754.337858:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.337868:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.337894:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760754.337926:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493766528:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760754.338141:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6118, 6118] 00000080:00200000:0.0:1789760754.338156:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.338163:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.338178:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25063424 i_blocks 49159 00000020:00200000:0.0:1789760754.338182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.338191:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.338222:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25063424, count: 4096 00000080:00200000:0.0:1789760754.338230:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25063424, bytes: 4096 00000080:00200000:0.0:1789760754.338255:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.338266:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.338268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25063424, 25067520) 0 0 00000080:00200000:0.0:1789760754.338273:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.338360:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6119, 6119] 00020000:00200000:0.0:1789760754.338372:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25063424 chunk: [25063424, 25067520) 25067520, 4096 00020000:00200000:0.0:1789760754.338377:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.338425:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6119, 6119] 00000008:00010000:0.0:1789760754.338446:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.338452:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760754.338464:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25063424, 25067520) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.338479:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25063424=0x17e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.338541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.338544:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25067520 00000020:00010000:0.0:1789760754.338554:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.338559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.338562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.338566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.338570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.338578:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.338585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.338602:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760754.338710:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 25063424<>25067520 00000100:00100000:1.0:1789760754.339164:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760754.339189:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493766656:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760754.375657:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973437 gen 3 00000100:00100000:1.0:1789760754.375702:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304dc00 x1876699493766656/t4294973438(4294973438) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760770 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760754.375749:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760754.375755:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760754.375765:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760754.375787:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760754.375813:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493766656:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760754.375848:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6119, 6119] 00000080:00200000:0.0:1789760754.375865:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.375873:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.375894:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25067520 i_blocks 49159 00000020:00200000:0.0:1789760754.375898:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.375908:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.375943:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25067520, count: 4096 00000080:00200000:0.0:1789760754.375950:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25067520, bytes: 4096 00000080:00200000:0.0:1789760754.375963:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.375972:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.375974:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25067520, 25071616) 0 0 00000080:00200000:0.0:1789760754.375978:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.375994:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6120, 6120] 00020000:00200000:0.0:1789760754.376002:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25067520 chunk: [25067520, 25071616) 25071616, 4096 00020000:00200000:0.0:1789760754.376007:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.376026:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6120, 6120] 00000008:00010000:0.0:1789760754.376048:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.376054:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760754.376066:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25067520, 25071616) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.376079:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25067520=0x17e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.376142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.376145:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25071616 00000020:00010000:0.0:1789760754.376154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.376158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.376161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.376164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.376167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.376175:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.376182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.376198:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760754.376486:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 25067520<>25071616 00000100:00100000:0.0:1789760754.376531:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760754.376549:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493766784:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760754.416933:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973439 gen 3 00000008:00010000:0.0:1789760754.417007:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760754.417018:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.417037:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.417061:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760754.417091:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493766784:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760754.417553:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6120, 6120] 00000080:00200000:1.0:1789760754.417571:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.417576:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.417591:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25071616 i_blocks 49159 00000020:00200000:1.0:1789760754.417594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.417605:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.417646:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25071616, count: 4096 00000080:00200000:1.0:1789760754.417653:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25071616, bytes: 4096 00000080:00200000:1.0:1789760754.417665:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.417673:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.417675:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25071616, 25075712) 0 0 00000080:00200000:1.0:1789760754.417679:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.417695:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6121, 6121] 00020000:00200000:1.0:1789760754.417704:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25071616 chunk: [25071616, 25075712) 25075712, 4096 00020000:00200000:1.0:1789760754.417708:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.417727:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6121, 6121] 00000008:00010000:1.0:1789760754.417744:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.417750:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760754.417762:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25071616, 25075712) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.417778:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25071616=0x17e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.417838:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.417841:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25075712 00000020:00010000:1.0:1789760754.417850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.417854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.417857:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.417861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.417864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.417875:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.417882:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.417898:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760754.418006:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 25071616<>25075712 00000100:00100000:1.0:1789760754.418055:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760754.418074:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493766912:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760754.483285:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973440 gen 3 00000008:00010000:1.0:1789760754.483338:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:1.0:1789760754.483346:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760754.483357:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760754.483385:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:1.0:1789760754.483446:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493766912:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760754.483823:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6121, 6121] 00000080:00200000:2.0:1789760754.483840:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.483847:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.483864:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25075712 i_blocks 49159 00000020:00200000:2.0:1789760754.483868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.483878:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.483911:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25075712, count: 4096 00000080:00200000:2.0:1789760754.483919:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25075712, bytes: 4096 00000080:00200000:2.0:1789760754.483928:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.483937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.483939:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25075712, 25079808) 0 0 00000080:00200000:2.0:1789760754.483944:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.483959:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6122, 6122] 00020000:00200000:2.0:1789760754.483967:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25075712 chunk: [25075712, 25079808) 25079808, 4096 00020000:00200000:2.0:1789760754.483971:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.483990:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6122, 6122] 00000008:00010000:2.0:1789760754.484006:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.484012:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760754.484022:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25075712, 25079808) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.484036:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25075712=0x17ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.484090:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.484093:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25079808 00000020:00010000:2.0:1789760754.484101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.484104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.484107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.484110:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.484112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.484120:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.484126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.484140:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760754.484250:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25075712<>25079808 00000100:00100000:3.0:1789760754.484322:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767040:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.518856:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973441 gen 3 00000008:00010000:2.0:1789760754.518914:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760754.518923:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.518935:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.519016:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760754.519047:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767040:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760754.519792:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6122, 6122] 00000080:00200000:1.0:1789760754.519813:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.519830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.519849:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25079808 i_blocks 49159 00000020:00200000:1.0:1789760754.519854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.519868:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.519902:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25079808, count: 4096 00000080:00200000:1.0:1789760754.519911:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25079808, bytes: 4096 00000080:00200000:1.0:1789760754.519927:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.519937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.519940:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25079808, 25083904) 0 0 00000080:00200000:1.0:1789760754.519944:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.519960:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6123, 6123] 00020000:00200000:1.0:1789760754.519971:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25079808 chunk: [25079808, 25083904) 25083904, 4096 00020000:00200000:1.0:1789760754.519975:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.519998:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6123, 6123] 00000008:00010000:1.0:1789760754.520017:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.520023:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760754.520036:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25079808, 25083904) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.520049:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25079808=0x17eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.520114:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.520117:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25083904 00000020:00010000:1.0:1789760754.520126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.520916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.520925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.520929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.520932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.520948:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.520956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.520983:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760754.521103:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 25079808<>25083904 00000100:00100000:0.0:1789760754.521790:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767168:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760754.569253:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973442 gen 3 00000008:00010000:0.0:1789760754.569321:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760754.569332:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.569346:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.569375:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760754.569460:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767168:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760754.569778:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6123, 6123] 00000080:00200000:2.0:1789760754.569802:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.569809:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.569961:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25083904 i_blocks 49159 00000020:00200000:2.0:1789760754.569966:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.569981:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.570028:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25083904, count: 4096 00000080:00200000:2.0:1789760754.570037:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25083904, bytes: 4096 00000080:00200000:2.0:1789760754.570050:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.570062:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.570064:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25083904, 25088000) 0 0 00000080:00200000:2.0:1789760754.570070:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.570091:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6124, 6124] 00020000:00200000:2.0:1789760754.570103:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25083904 chunk: [25083904, 25088000) 25088000, 4096 00020000:00200000:2.0:1789760754.570110:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.570136:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6124, 6124] 00000008:00010000:2.0:1789760754.570159:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.570168:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760754.570181:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25083904, 25088000) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.570210:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25083904=0x17ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.570282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.570285:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25088000 00000020:00010000:2.0:1789760754.570293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.570296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.570299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.570304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.570306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.570314:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.570321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.570337:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760754.570473:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25083904<>25088000 00000100:00100000:3.0:1789760754.570569:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493767296:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.625642:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973443 gen 3 00000008:00010000:2.0:1789760754.625699:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760754.625707:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.625789:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.625814:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760754.625903:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493767296:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760754.626072:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6124, 6124] 00000080:00200000:0.0:1789760754.626089:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760754.626095:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760754.626111:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25088000 i_blocks 49159 00000020:00200000:0.0:1789760754.626116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760754.626128:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760754.626163:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25088000, count: 4096 00000080:00200000:0.0:1789760754.626169:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25088000, bytes: 4096 00000080:00200000:0.0:1789760754.626179:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760754.626187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760754.626190:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25088000, 25092096) 0 0 00000080:00200000:0.0:1789760754.626194:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760754.626211:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6125, 6125] 00020000:00200000:0.0:1789760754.626220:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25088000 chunk: [25088000, 25092096) 25092096, 4096 00020000:00200000:0.0:1789760754.626224:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760754.626257:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6125, 6125] 00000008:00010000:0.0:1789760754.626275:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760754.626294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760754.626306:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25088000, 25092096) append 0 append lockpos 0 00000080:00200000:0.0:1789760754.626321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25088000=0x17ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760754.626379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760754.626382:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25092096 00000020:00010000:0.0:1789760754.626390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760754.626566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.626570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760754.626573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760754.626576:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760754.626585:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760754.626592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760754.626609:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760754.626988:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 25088000<>25092096 00000100:00100000:1.0:1789760754.627046:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760754.627631:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767424:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760754.659612:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973444 gen 3 00000008:00010000:1.0:1789760754.659672:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760754.659681:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760754.659693:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760754.659773:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760754.659822:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760754.660489:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6125, 6125] 00000080:00200000:2.0:1789760754.660507:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.660513:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.660530:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25092096 i_blocks 49159 00000020:00200000:2.0:1789760754.660534:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.660545:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.660580:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25092096, count: 4096 00000080:00200000:2.0:1789760754.660588:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25092096, bytes: 4096 00000080:00200000:2.0:1789760754.660599:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.660608:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.660611:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25092096, 25096192) 0 0 00000080:00200000:2.0:1789760754.660615:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.660633:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6126, 6126] 00020000:00200000:2.0:1789760754.660641:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25092096 chunk: [25092096, 25096192) 25096192, 4096 00020000:00200000:2.0:1789760754.660645:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.660665:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6126, 6126] 00000008:00010000:2.0:1789760754.660682:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.660689:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760754.660701:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25092096, 25096192) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.660785:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25092096=0x17ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.661132:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.661135:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25096192 00000020:00010000:2.0:1789760754.661146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.661150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.661153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.661157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.661160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.661169:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.661177:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.661199:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760754.661590:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25092096<>25096192 00000100:00100000:3.0:1789760754.661651:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760754.662019:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493767552:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.705284:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973445 gen 3 00000008:00010000:2.0:1789760754.705335:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760754.705345:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.705358:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.705506:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6126, 6126] 00000008:00010000:2.0:1789760754.705510:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000080:00200000:3.0:1789760754.705527:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.705535:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:2.0:1789760754.705542:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493767552:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.705551:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25096192 i_blocks 49159 00000020:00200000:3.0:1789760754.705555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.705566:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.705603:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25096192, count: 4096 00000080:00200000:3.0:1789760754.705611:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25096192, bytes: 4096 00000080:00200000:3.0:1789760754.705621:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.705630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.705632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25096192, 25100288) 0 0 00000080:00200000:3.0:1789760754.705636:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.705654:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6127, 6127] 00020000:00200000:3.0:1789760754.705664:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25096192 chunk: [25096192, 25100288) 25100288, 4096 00020000:00200000:3.0:1789760754.705668:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.705689:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6127, 6127] 00000008:00010000:3.0:1789760754.705707:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.705713:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.705724:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25096192, 25100288) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.705877:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25096192=0x17ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.705938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.705941:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25100288 00000020:00010000:3.0:1789760754.705951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.705954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.705957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.705960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.705962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.705971:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.705977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.705991:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760754.706093:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25096192<>25100288 00000100:00100000:3.0:1789760754.706139:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760754.706155:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767680:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.738623:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973446 gen 3 00000008:00010000:2.0:1789760754.738670:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760754.738678:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.738688:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.738709:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760754.738747:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767680:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.738973:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6127, 6127] 00000080:00200000:3.0:1789760754.738992:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.738998:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.739014:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25100288 i_blocks 49159 00000020:00200000:3.0:1789760754.739018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.739028:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.739064:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25100288, count: 4096 00000080:00200000:3.0:1789760754.739071:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25100288, bytes: 4096 00000080:00200000:3.0:1789760754.739082:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.739091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.739094:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25100288, 25104384) 0 0 00000080:00200000:3.0:1789760754.739098:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.739116:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6128, 6128] 00020000:00200000:3.0:1789760754.739124:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25100288 chunk: [25100288, 25104384) 25104384, 4096 00020000:00200000:3.0:1789760754.739129:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.739150:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6128, 6128] 00000008:00010000:3.0:1789760754.739169:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.739176:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.739189:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25100288, 25104384) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.739203:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25100288=0x17f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.739271:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.739274:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25104384 00000020:00010000:3.0:1789760754.739294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.739298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.739301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.739306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.739309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.739316:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.739324:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.739340:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760754.739603:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25100288<>25104384 00000100:00100000:3.0:1789760754.739650:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767808:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.768288:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973447 gen 3 00000008:00010000:2.0:1789760754.768341:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760754.768349:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.768361:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.768446:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6128, 6128] 00000008:00010000:2.0:1789760754.768449:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000080:00200000:1.0:1789760754.768465:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.768472:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.768489:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25104384 i_blocks 49159 00000020:00200000:1.0:1789760754.768494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.768504:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.768541:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25104384, count: 4096 00000080:00200000:1.0:1789760754.768550:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25104384, bytes: 4096 00000080:00200000:1.0:1789760754.768560:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.768569:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.768571:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25104384, 25108480) 0 0 00000080:00200000:1.0:1789760754.768576:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.768594:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6129, 6129] 00020000:00200000:1.0:1789760754.768602:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25104384 chunk: [25104384, 25108480) 25108480, 4096 00020000:00200000:1.0:1789760754.768606:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.768625:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6129, 6129] 00000008:00010000:1.0:1789760754.768642:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.768648:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760754.768659:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25104384, 25108480) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.768673:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25104384=0x17f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.768807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.768811:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25108480 00000020:00010000:1.0:1789760754.768892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.768897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.768900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.768904:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.768907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.768918:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.768924:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.768944:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760754.769060:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 25104384<>25108480 00000100:00100000:0.0:1789760754.769143:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767936:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.769520:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493767808:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760754.825193:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973448 gen 3 00000008:00010000:0.0:1789760754.825257:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92cc0 00000020:00200000:0.0:1789760754.825266:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760754.825277:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760754.825381:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92cc0 00000100:00100000:0.0:1789760754.825445:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493767936:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.825546:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6129, 6129] 00000080:00200000:3.0:1789760754.825562:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.825569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.825586:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25108480 i_blocks 49159 00000020:00200000:3.0:1789760754.825590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.825600:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.825636:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25108480, count: 4096 00000080:00200000:3.0:1789760754.825643:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25108480, bytes: 4096 00000080:00200000:3.0:1789760754.825653:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.825662:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.825664:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25108480, 25112576) 0 0 00000080:00200000:3.0:1789760754.825669:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.825688:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6130, 6130] 00020000:00200000:3.0:1789760754.825697:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25108480 chunk: [25108480, 25112576) 25112576, 4096 00020000:00200000:3.0:1789760754.825702:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.825786:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6130, 6130] 00000008:00010000:3.0:1789760754.825807:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.825813:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.825888:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25108480, 25112576) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.825904:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25108480=0x17f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.825967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.825970:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25112576 00000020:00010000:3.0:1789760754.825978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.825982:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.825985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.825989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.825992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.826000:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.826007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.826024:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760754.826204:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25108480<>25112576 00000100:00100000:2.0:1789760754.826514:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768064:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.870115:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973449 gen 3 00000008:00010000:2.0:1789760754.870164:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760754.870172:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.870182:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.870203:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760754.870321:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768064:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.870723:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6130, 6130] 00000080:00200000:3.0:1789760754.870740:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.870746:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.870762:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25112576 i_blocks 49159 00000020:00200000:3.0:1789760754.870766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.870776:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.870810:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25112576, count: 4096 00000080:00200000:3.0:1789760754.870817:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25112576, bytes: 4096 00000080:00200000:3.0:1789760754.870829:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.870839:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.870841:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25112576, 25116672) 0 0 00000080:00200000:3.0:1789760754.870846:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.870863:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6131, 6131] 00020000:00200000:3.0:1789760754.870872:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25112576 chunk: [25112576, 25116672) 25116672, 4096 00020000:00200000:3.0:1789760754.870876:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.870894:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6131, 6131] 00000008:00010000:3.0:1789760754.870913:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.870919:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.870931:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25112576, 25116672) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.870946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25112576=0x17f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.871004:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.871007:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25116672 00000020:00010000:3.0:1789760754.871016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.871020:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.871023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.871026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.871029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.871037:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.871044:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.871060:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760754.871170:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25112576<>25116672 00000100:00100000:2.0:1789760754.871553:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760754.871575:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768192:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.911345:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973450 gen 3 00000008:00010000:2.0:1789760754.911423:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760754.911432:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.911444:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.911466:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760754.911496:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768192:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.911600:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6131, 6131] 00000080:00200000:3.0:1789760754.911617:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.911624:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.911640:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25116672 i_blocks 49159 00000020:00200000:3.0:1789760754.911644:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.911654:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.911688:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25116672, count: 4096 00000080:00200000:3.0:1789760754.911695:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25116672, bytes: 4096 00000080:00200000:3.0:1789760754.911705:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.911724:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.911726:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25116672, 25120768) 0 0 00000080:00200000:3.0:1789760754.911732:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.911749:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6132, 6132] 00020000:00200000:3.0:1789760754.911757:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25116672 chunk: [25116672, 25120768) 25120768, 4096 00020000:00200000:3.0:1789760754.911761:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.911781:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6132, 6132] 00000008:00010000:3.0:1789760754.911799:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.911805:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.911827:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25116672, 25120768) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.911843:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25116672=0x17f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.911904:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.911907:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25120768 00000020:00010000:3.0:1789760754.911915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.911919:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.911922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.911926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.911929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.911936:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.911943:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.911957:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760754.912057:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25116672<>25120768 00000100:00100000:2.0:1789760754.912319:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760754.912338:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768320:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760754.924670:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760754.947337:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973451 gen 3 00000008:00010000:2.0:1789760754.947419:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760754.947428:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.947440:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.947461:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760754.947491:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768320:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.947591:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6132, 6132] 00000080:00200000:3.0:1789760754.947607:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.947613:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.947629:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25120768 i_blocks 49159 00000020:00200000:3.0:1789760754.947633:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.947642:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.947678:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25120768, count: 4096 00000080:00200000:3.0:1789760754.947686:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25120768, bytes: 4096 00000080:00200000:3.0:1789760754.947694:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.947703:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.947706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25120768, 25124864) 0 0 00000080:00200000:3.0:1789760754.947717:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.947734:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6133, 6133] 00020000:00200000:3.0:1789760754.947743:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25120768 chunk: [25120768, 25124864) 25124864, 4096 00020000:00200000:3.0:1789760754.947747:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.947767:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6133, 6133] 00000008:00010000:3.0:1789760754.947783:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.947789:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.947799:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25120768, 25124864) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.947813:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25120768=0x17f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.947869:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.947872:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25124864 00000020:00010000:3.0:1789760754.947882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.947886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.947890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.947895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.947898:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.947908:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.947915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.947931:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760754.948031:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25120768<>25124864 00000100:00100000:2.0:1789760754.948122:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760754.948140:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768448:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760754.987499:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973452 gen 3 00000008:00010000:2.0:1789760754.987551:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760754.987560:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.987571:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.987591:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760754.987620:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768448:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760754.987730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6133, 6133] 00000080:00200000:3.0:1789760754.987746:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.987752:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.987766:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25124864 i_blocks 49159 00000020:00200000:3.0:1789760754.987769:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.987779:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.987813:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25124864, count: 4096 00000080:00200000:3.0:1789760754.987820:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25124864, bytes: 4096 00000080:00200000:3.0:1789760754.987830:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.987839:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.987841:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25124864, 25128960) 0 0 00000080:00200000:3.0:1789760754.987845:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.987862:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6134, 6134] 00020000:00200000:3.0:1789760754.987870:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25124864 chunk: [25124864, 25128960) 25128960, 4096 00020000:00200000:3.0:1789760754.987874:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.987893:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6134, 6134] 00000008:00010000:3.0:1789760754.987912:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.987918:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760754.987930:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25124864, 25128960) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.987946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25124864=0x17f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.988005:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.988008:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25128960 00000020:00010000:3.0:1789760754.988017:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.988020:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.988024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.988027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.988030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.988038:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.988045:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.988060:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760754.988161:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 25124864<>25128960 00000100:00100000:2.0:1789760754.988257:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760754.988274:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768576:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760755.040350:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760755.040367:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.040379:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.040479:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760755.040512:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768576:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.040586:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6134, 6134] 00000080:00200000:3.0:1789760755.040605:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.040611:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.040627:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25128960 i_blocks 49159 00000020:00200000:3.0:1789760755.040632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.040643:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.040682:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25128960, count: 4096 00000080:00200000:3.0:1789760755.040691:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25128960, bytes: 4096 00000080:00200000:3.0:1789760755.040701:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.040710:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.040712:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25128960, 25133056) 0 0 00000080:00200000:3.0:1789760755.040717:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.040734:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6135, 6135] 00020000:00200000:3.0:1789760755.040742:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25128960 chunk: [25128960, 25133056) 25133056, 4096 00020000:00200000:3.0:1789760755.040746:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.040766:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6135, 6135] 00000008:00010000:3.0:1789760755.040785:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.040791:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.040802:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25128960, 25133056) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.040873:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25128960=0x17f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.040935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.040938:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25133056 00000020:00010000:3.0:1789760755.040948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.040952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.040955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.040958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.040962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.040970:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.040977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.040994:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760755.041221:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25128960<>25133056 00000100:00100000:2.0:1789760755.041310:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768704:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.081018:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973454 gen 3 00000008:00010000:2.0:1789760755.081088:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760755.081096:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.081105:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.081125:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760755.081153:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493768704:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.081346:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6135, 6135] 00000080:00200000:3.0:1789760755.081365:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.081372:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.081388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25133056 i_blocks 49159 00000020:00200000:3.0:1789760755.081411:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.081422:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.081457:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25133056, count: 4096 00000080:00200000:3.0:1789760755.081465:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25133056, bytes: 4096 00000080:00200000:3.0:1789760755.081475:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.081484:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.081487:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25133056, 25137152) 0 0 00000080:00200000:3.0:1789760755.081495:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.081510:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6136, 6136] 00020000:00200000:3.0:1789760755.081517:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25133056 chunk: [25133056, 25137152) 25137152, 4096 00020000:00200000:3.0:1789760755.081524:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.081543:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6136, 6136] 00000008:00010000:3.0:1789760755.081560:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.081566:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.081576:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25133056, 25137152) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.081589:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25133056=0x17f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.081636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.081639:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25137152 00000020:00010000:3.0:1789760755.081648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.081651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.081654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.081658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.081660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.081668:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.081674:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.081696:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760755.081926:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25133056<>25137152 00000100:00100000:2.0:1789760755.081997:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768832:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.122339:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973455 gen 3 00000008:00010000:2.0:1789760755.122426:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760755.122435:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.122447:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.122469:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760755.122498:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493768832:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.122591:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6136, 6136] 00000080:00200000:0.0:1789760755.122607:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.122614:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.122629:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25137152 i_blocks 49159 00000020:00200000:0.0:1789760755.122632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.122642:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.122676:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25137152, count: 4096 00000080:00200000:0.0:1789760755.122683:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25137152, bytes: 4096 00000080:00200000:0.0:1789760755.122694:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.122704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.122706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25137152, 25141248) 0 0 00000080:00200000:0.0:1789760755.122710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.122725:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6137, 6137] 00020000:00200000:0.0:1789760755.122734:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25137152 chunk: [25137152, 25141248) 25141248, 4096 00020000:00200000:0.0:1789760755.122738:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.122768:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6137, 6137] 00000008:00010000:0.0:1789760755.122787:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.122794:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.122805:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25137152, 25141248) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.122829:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25137152=0x17f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.122886:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.122888:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25141248 00000020:00010000:0.0:1789760755.122897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.122901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.122904:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.122908:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.122913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.122921:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.122931:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.122947:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760755.123051:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 25137152<>25141248 00000100:00100000:0.0:1789760755.123101:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760755.123118:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493768960:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760755.159363:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760755.159381:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760755.159424:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760755.159500:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760755.159531:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493768960:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760755.160257:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6137, 6137] 00000080:00200000:2.0:1789760755.160276:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.160282:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.160356:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25141248 i_blocks 49159 00000020:00200000:2.0:1789760755.160361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.160374:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.160442:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25141248, count: 4096 00000080:00200000:2.0:1789760755.160450:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25141248, bytes: 4096 00000080:00200000:2.0:1789760755.160463:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.160473:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.160475:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25141248, 25145344) 0 0 00000080:00200000:2.0:1789760755.160480:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.160498:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6138, 6138] 00020000:00200000:2.0:1789760755.160508:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25141248 chunk: [25141248, 25145344) 25145344, 4096 00020000:00200000:2.0:1789760755.160512:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.160533:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6138, 6138] 00000008:00010000:2.0:1789760755.160553:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.160559:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760755.160571:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25141248, 25145344) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.160587:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25141248=0x17fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.160645:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.160648:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25145344 00000020:00010000:2.0:1789760755.160656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.160659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.160662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.160666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.160669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.160677:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.160684:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.160703:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760755.160944:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25141248<>25145344 00000100:00100000:2.0:1789760755.160988:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760755.161010:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493769088:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.219503:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973457 gen 3 00000008:00010000:2.0:1789760755.219570:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760755.219579:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.219590:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.219615:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760755.219645:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493769088:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.220059:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6138, 6138] 00000080:00200000:3.0:1789760755.220076:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.220083:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.220101:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25145344 i_blocks 49159 00000020:00200000:3.0:1789760755.220106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.220118:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.220152:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25145344, count: 4096 00000080:00200000:3.0:1789760755.220160:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25145344, bytes: 4096 00000080:00200000:3.0:1789760755.220173:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.220182:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.220184:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25145344, 25149440) 0 0 00000080:00200000:3.0:1789760755.220189:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.220205:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6139, 6139] 00020000:00200000:3.0:1789760755.220214:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25145344 chunk: [25145344, 25149440) 25149440, 4096 00020000:00200000:3.0:1789760755.220219:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.220250:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6139, 6139] 00000008:00010000:3.0:1789760755.220269:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.220275:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.220352:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25145344, 25149440) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.220367:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25145344=0x17fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.221377:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.221384:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25149440 00000020:00010000:3.0:1789760755.221885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.221891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.221895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.221899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.221902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.221915:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.221924:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.221952:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760755.222085:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25145344<>25149440 00000100:00100000:3.0:1789760755.222139:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760755.222163:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493769216:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.328528:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973458 gen 3 00000008:00010000:2.0:1789760755.330391:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760755.330420:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.330437:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.330467:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760755.330507:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493769216:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.331368:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6139, 6139] 00000080:00200000:3.0:1789760755.331387:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.331417:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.331434:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25149440 i_blocks 49159 00000020:00200000:3.0:1789760755.331439:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.331451:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.331497:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25149440, count: 4096 00000080:00200000:3.0:1789760755.331505:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25149440, bytes: 4096 00000080:00200000:3.0:1789760755.331520:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.331531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.331533:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25149440, 25153536) 0 0 00000080:00200000:3.0:1789760755.331539:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.331557:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6140, 6140] 00020000:00200000:3.0:1789760755.331566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25149440 chunk: [25149440, 25153536) 25153536, 4096 00020000:00200000:3.0:1789760755.331570:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.331592:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6140, 6140] 00000008:00010000:3.0:1789760755.331612:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.331618:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.331631:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25149440, 25153536) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.331646:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25149440=0x17fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.331710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.331713:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25153536 00000020:00010000:3.0:1789760755.331721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.331724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.331727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.331730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.331733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.331741:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.331749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.331767:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760755.331877:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25149440<>25153536 00000100:00100000:2.0:1789760755.337018:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760755.337063:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493769344:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760755.383414:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760755.383432:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.383445:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.383472:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760755.383504:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493769344:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.383607:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6140, 6140] 00000080:00200000:0.0:1789760755.383623:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.383629:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.383644:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25153536 i_blocks 49159 00000020:00200000:0.0:1789760755.383648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.383658:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.383694:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25153536, count: 4096 00000080:00200000:0.0:1789760755.383701:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25153536, bytes: 4096 00000080:00200000:0.0:1789760755.383714:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.383724:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.383726:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25153536, 25157632) 0 0 00000080:00200000:0.0:1789760755.383731:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.383746:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6141, 6141] 00020000:00200000:0.0:1789760755.383755:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25153536 chunk: [25153536, 25157632) 25157632, 4096 00020000:00200000:0.0:1789760755.383759:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.383779:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6141, 6141] 00000008:00010000:0.0:1789760755.383798:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.383804:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.383815:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25153536, 25157632) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.383830:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25153536=0x17fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.383889:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.383892:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25157632 00000020:00010000:0.0:1789760755.383901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.383905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.383908:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.383912:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.383915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.383922:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.383929:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.383959:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760755.384060:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 25153536<>25157632 00000100:00100000:1.0:1789760755.384290:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769472:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760755.434979:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973460 gen 3 00000008:00010000:1.0:1789760755.435057:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760755.435065:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.435075:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.435173:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760755.435202:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769472:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.435563:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6141, 6141] 00000080:00200000:0.0:1789760755.435583:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.435590:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.435608:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25157632 i_blocks 49159 00000020:00200000:0.0:1789760755.435612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.435624:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.435660:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25157632, count: 4096 00000080:00200000:0.0:1789760755.435668:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25157632, bytes: 4096 00000080:00200000:0.0:1789760755.435749:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.435758:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.435761:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25157632, 25161728) 0 0 00000080:00200000:0.0:1789760755.435766:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.435782:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6142, 6142] 00020000:00200000:0.0:1789760755.435791:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25157632 chunk: [25157632, 25161728) 25161728, 4096 00020000:00200000:0.0:1789760755.435796:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.435822:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6142, 6142] 00000008:00010000:0.0:1789760755.435841:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.435848:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.435861:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25157632, 25161728) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.435876:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25157632=0x17fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.435934:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.435937:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25161728 00000020:00010000:0.0:1789760755.435945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.435948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.435952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.435955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.435958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.435966:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.435973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.435988:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760755.436092:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 25157632<>25161728 00000100:00100000:1.0:1789760755.436170:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493769600:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760755.464804:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973461 gen 3 00000008:00010000:1.0:1789760755.464856:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760755.464865:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.464877:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.464899:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760755.464931:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493769600:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.465850:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6142, 6142] 00000080:00200000:3.0:1789760755.465875:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.465891:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.465908:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25161728 i_blocks 49159 00000020:00200000:3.0:1789760755.465912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.465923:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.465956:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25161728, count: 4096 00000080:00200000:3.0:1789760755.465964:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25161728, bytes: 4096 00000080:00200000:3.0:1789760755.465977:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.465987:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.465989:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25161728, 25165824) 0 0 00000080:00200000:3.0:1789760755.465993:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.466009:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6143, 6143] 00020000:00200000:3.0:1789760755.466018:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25161728 chunk: [25161728, 25165824) 25165824, 4096 00020000:00200000:3.0:1789760755.466023:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.466043:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6143, 6143] 00000008:00010000:3.0:1789760755.466062:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.466069:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.466082:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25161728, 25165824) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.466096:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25161728=0x17ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.466165:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.466168:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25165824 00000020:00010000:3.0:1789760755.466177:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.466181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.466184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.466837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.466843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.466855:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.466863:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.467091:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760755.468309:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25161728<>25165824 00000100:00100000:2.0:1789760755.468695:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760755.468856:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493769728:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.517969:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973462 gen 3 00000008:00010000:2.0:1789760755.518023:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760755.518032:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.518043:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.518074:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760755.518104:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493769728:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.518267:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6143, 6143] 00000080:00200000:0.0:1789760755.518300:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.518308:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.518323:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25165824 i_blocks 49159 00000020:00200000:0.0:1789760755.518327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.518336:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.518371:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25165824, count: 4096 00000080:00200000:0.0:1789760755.518378:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25165824, bytes: 4096 00000080:00200000:0.0:1789760755.518388:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.518427:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.518430:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25165824, 25169920) 0 0 00000080:00200000:0.0:1789760755.518434:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.518450:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6144, 6144] 00020000:00200000:0.0:1789760755.518460:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25165824 chunk: [25165824, 25169920) 25169920, 4096 00020000:00200000:0.0:1789760755.518465:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.518486:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6144, 6144] 00000008:00010000:0.0:1789760755.518505:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.518512:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.518525:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25165824, 25169920) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.518539:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25165824=0x1800000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.518600:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.518603:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25169920 00000020:00010000:0.0:1789760755.518612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.518616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.518619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.518622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.518625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.518633:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.518641:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.518658:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760755.519127:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 25165824<>25169920 00000100:00100000:1.0:1789760755.519503:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760755.567919:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973463 gen 3 00000008:00010000:1.0:1789760755.567973:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760755.567983:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.567994:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.568018:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760755.568049:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769856:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.568428:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6144, 6144] 00000080:00200000:0.0:1789760755.568447:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.568454:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.568472:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25169920 i_blocks 49159 00000020:00200000:0.0:1789760755.568476:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.568488:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.568527:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25169920, count: 4096 00000080:00200000:0.0:1789760755.568535:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25169920, bytes: 4096 00000080:00200000:0.0:1789760755.568546:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.568555:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.568558:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25169920, 25174016) 0 0 00000080:00200000:0.0:1789760755.568564:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.568580:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6145, 6145] 00020000:00200000:0.0:1789760755.568589:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25169920 chunk: [25169920, 25174016) 25174016, 4096 00020000:00200000:0.0:1789760755.568594:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.568613:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6145, 6145] 00000008:00010000:0.0:1789760755.568632:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.568639:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.568651:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25169920, 25174016) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.568666:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25169920=0x1801000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.568737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.568740:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25174016 00000020:00010000:0.0:1789760755.568750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.568754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.568757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.568761:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.568764:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.568772:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.568779:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.568796:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760755.568897:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 25169920<>25174016 00000100:00100000:1.0:1789760755.569056:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760755.569079:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760755.625064:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973464 gen 3 00000008:00010000:1.0:1789760755.625114:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760755.625122:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.625132:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.625210:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760755.625249:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493769984:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.625479:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6145, 6145] 00000080:00200000:0.0:1789760755.625495:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.625502:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.625518:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25174016 i_blocks 51207 00000020:00200000:0.0:1789760755.625522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.625533:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.625569:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25174016, count: 4096 00000080:00200000:0.0:1789760755.625576:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25174016, bytes: 4096 00000080:00200000:0.0:1789760755.625587:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.625596:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.625598:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25174016, 25178112) 0 0 00000080:00200000:0.0:1789760755.625604:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.625620:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6146, 6146] 00020000:00200000:0.0:1789760755.625629:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25174016 chunk: [25174016, 25178112) 25178112, 4096 00020000:00200000:0.0:1789760755.625634:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.625654:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6146, 6146] 00000008:00010000:0.0:1789760755.625672:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.625678:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.625690:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25174016, 25178112) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.626012:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25174016=0x1802000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.626074:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.626077:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25178112 00000020:00010000:0.0:1789760755.626086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.626090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.626094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.626098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.626101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.626110:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.626117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.626139:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760755.626254:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 25174016<>25178112 00000100:00100000:1.0:1789760755.626345:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493770112:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760755.690304:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760755.690322:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.690334:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.690429:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760755.690470:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493770112:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.691420:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6146, 6146] 00000080:00200000:3.0:1789760755.691440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.691447:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.691463:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25178112 i_blocks 51207 00000020:00200000:3.0:1789760755.691467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.691479:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.691586:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25178112, count: 4096 00000080:00200000:3.0:1789760755.691595:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25178112, bytes: 4096 00000080:00200000:3.0:1789760755.691608:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.691616:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.691619:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25178112, 25182208) 0 0 00000080:00200000:3.0:1789760755.691624:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.691642:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6147, 6147] 00020000:00200000:3.0:1789760755.691651:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25178112 chunk: [25178112, 25182208) 25182208, 4096 00020000:00200000:3.0:1789760755.691655:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.691673:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6147, 6147] 00000008:00010000:3.0:1789760755.691686:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.691692:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.691704:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25178112, 25182208) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.691719:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25178112=0x1803000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.691856:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.691860:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25182208 00000020:00010000:3.0:1789760755.691868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.691872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.691875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.691878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.691881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.691890:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.691897:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.691921:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760755.692387:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25178112<>25182208 00000100:00100000:3.0:1789760755.692479:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760755.692632:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770240:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.728110:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973466 gen 3 00000008:00010000:2.0:1789760755.728177:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760755.728186:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.728271:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.728383:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760755.728509:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770240:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.728605:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6147, 6147] 00000080:00200000:3.0:1789760755.728622:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.728629:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.728645:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25182208 i_blocks 51207 00000020:00200000:3.0:1789760755.728649:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.728669:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.728705:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25182208, count: 4096 00000080:00200000:3.0:1789760755.728713:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25182208, bytes: 4096 00000080:00200000:3.0:1789760755.728723:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.728732:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.728735:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25182208, 25186304) 0 0 00000080:00200000:3.0:1789760755.728910:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.728928:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6148, 6148] 00020000:00200000:3.0:1789760755.728937:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25182208 chunk: [25182208, 25186304) 25186304, 4096 00020000:00200000:3.0:1789760755.728942:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.728964:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6148, 6148] 00000008:00010000:3.0:1789760755.728982:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.728988:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.729000:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25182208, 25186304) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.729014:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25182208=0x1804000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.729073:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.729077:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25186304 00000020:00010000:3.0:1789760755.729085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.729088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.729091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.729095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.729098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.729106:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.729113:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.729129:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760755.729365:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25182208<>25186304 00000100:00100000:2.0:1789760755.729502:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770368:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760755.785645:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760755.785726:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.785847:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.785873:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760755.785909:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770368:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.786026:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6148, 6148] 00000080:00200000:3.0:1789760755.786044:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.786049:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.786063:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25186304 i_blocks 51207 00000020:00200000:3.0:1789760755.786068:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.786077:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.786111:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25186304, count: 4096 00000080:00200000:3.0:1789760755.786120:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25186304, bytes: 4096 00000080:00200000:3.0:1789760755.786132:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.786141:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.786144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25186304, 25190400) 0 0 00000080:00200000:3.0:1789760755.786149:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.786165:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6149, 6149] 00020000:00200000:3.0:1789760755.786174:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25186304 chunk: [25186304, 25190400) 25190400, 4096 00020000:00200000:3.0:1789760755.786179:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.786198:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6149, 6149] 00000008:00010000:3.0:1789760755.786217:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.786234:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.786246:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25186304, 25190400) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.786261:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25186304=0x1805000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.786328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.786332:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25190400 00000020:00010000:3.0:1789760755.786340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.786344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.786348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.786352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.786354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.786362:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.786368:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.786384:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760755.786915:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25186304<>25190400 00000100:00100000:2.0:1789760755.787024:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760755.787043:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770496:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.825612:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973467 gen 3 00000100:00100000:2.0:1789760755.825656:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b800 x1876699493770496/t4294973468(4294973468) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760771 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760755.825712:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760755.825720:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.825741:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.825762:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760755.825787:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770496:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.826049:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6149, 6149] 00000080:00200000:3.0:1789760755.826065:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.826071:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.826088:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25190400 i_blocks 51207 00000020:00200000:3.0:1789760755.826092:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.826102:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.826136:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25190400, count: 4096 00000080:00200000:3.0:1789760755.826144:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25190400, bytes: 4096 00000080:00200000:3.0:1789760755.826155:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.826164:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.826167:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25190400, 25194496) 0 0 00000080:00200000:3.0:1789760755.826172:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.826191:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6150, 6150] 00020000:00200000:3.0:1789760755.826200:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25190400 chunk: [25190400, 25194496) 25194496, 4096 00020000:00200000:3.0:1789760755.826205:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.826238:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6150, 6150] 00000008:00010000:3.0:1789760755.826259:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.826266:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.826322:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25190400, 25194496) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.826339:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25190400=0x1806000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.826435:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.826438:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25194496 00000020:00010000:3.0:1789760755.826447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.826450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.826454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.826457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.826460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.826469:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.826477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.826495:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760755.826613:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25190400<>25194496 00000100:00100000:2.0:1789760755.826844:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770624:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.872474:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973469 gen 3 00000008:00010000:2.0:1789760755.872618:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760755.872628:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.872637:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.872659:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760755.872686:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493770624:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760755.873111:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6150, 6150] 00000080:00200000:3.0:1789760755.873127:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.873133:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.873149:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25194496 i_blocks 51207 00000020:00200000:3.0:1789760755.873153:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.873163:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.873199:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25194496, count: 4096 00000080:00200000:3.0:1789760755.873207:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25194496, bytes: 4096 00000080:00200000:3.0:1789760755.873217:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.873238:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.873242:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25194496, 25198592) 0 0 00000080:00200000:3.0:1789760755.873246:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.873262:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6151, 6151] 00020000:00200000:3.0:1789760755.873270:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25194496 chunk: [25194496, 25198592) 25198592, 4096 00020000:00200000:3.0:1789760755.873338:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.873359:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6151, 6151] 00000008:00010000:3.0:1789760755.873376:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.873382:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760755.873418:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25194496, 25198592) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.873434:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25194496=0x1807000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.874097:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.874101:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25198592 00000020:00010000:3.0:1789760755.874112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.874116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.874119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.874123:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.874126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.874139:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.874147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.874169:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760755.876860:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25194496<>25198592 00000100:00100000:3.0:1789760755.876972:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493770752:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760755.914192:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973470 gen 3 00000008:00010000:2.0:1789760755.914251:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760755.914259:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.914271:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.914293:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760755.914320:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493770752:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760755.917257:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6151, 6151] 00000080:00200000:0.0:1789760755.917358:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.917366:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.917387:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25198592 i_blocks 51207 00000020:00200000:0.0:1789760755.917428:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.917519:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.917566:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25198592, count: 4096 00000080:00200000:0.0:1789760755.917577:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25198592, bytes: 4096 00000080:00200000:0.0:1789760755.917593:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.917604:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.917607:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25198592, 25202688) 0 0 00000080:00200000:0.0:1789760755.917611:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.917631:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6152, 6152] 00020000:00200000:0.0:1789760755.917642:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25198592 chunk: [25198592, 25202688) 25202688, 4096 00020000:00200000:0.0:1789760755.917648:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.918146:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6152, 6152] 00000008:00010000:0.0:1789760755.918173:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.918179:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760755.918193:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25198592, 25202688) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.918211:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25198592=0x1808000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.918379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.918383:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25202688 00000020:00010000:0.0:1789760755.918614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.918620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.918623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.918628:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.918631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.918643:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.920205:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.920259:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760755.920624:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 25198592<>25202688 00000100:00100000:1.0:1789760755.920973:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760755.921062:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493770880:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760755.949143:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760755.960120:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973471 gen 3 00000008:00010000:1.0:1789760755.960176:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760755.960186:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.960199:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.960290:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760755.960323:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493770880:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760755.960530:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6152, 6152] 00000080:00200000:2.0:1789760755.960549:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.960554:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.960573:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25202688 i_blocks 51207 00000020:00200000:2.0:1789760755.960576:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.960588:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.960684:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25202688, count: 4096 00000080:00200000:2.0:1789760755.960693:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25202688, bytes: 4096 00000080:00200000:2.0:1789760755.960772:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.960896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.960900:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25202688, 25206784) 0 0 00000080:00200000:2.0:1789760755.960906:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.960924:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6153, 6153] 00020000:00200000:2.0:1789760755.960933:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25202688 chunk: [25202688, 25206784) 25206784, 4096 00020000:00200000:2.0:1789760755.960937:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.960959:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6153, 6153] 00000008:00010000:2.0:1789760755.960978:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.960985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760755.960996:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25202688, 25206784) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.961011:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25202688=0x1809000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.961071:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.961073:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25206784 00000020:00010000:2.0:1789760755.961082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.961085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.961088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.961092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.961094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.961102:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.961109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.961127:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760755.961240:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25202688<>25206784 00000100:00100000:3.0:1789760755.961375:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771008:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760756.065487:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973472 gen 3 00000008:00010000:2.0:1789760756.065540:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:2.0:1789760756.065548:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760756.065559:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.065910:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6153, 6153] 00000080:00200000:1.0:1789760756.065928:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.065934:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.065950:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25206784 i_blocks 51207 00000020:00200000:1.0:1789760756.065954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.065967:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.066002:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25206784, count: 4096 00000080:00200000:1.0:1789760756.066011:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25206784, bytes: 4096 00000080:00200000:1.0:1789760756.066023:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.066032:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.066035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25206784, 25210880) 0 0 00000080:00200000:1.0:1789760756.066040:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.066058:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6154, 6154] 00020000:00200000:1.0:1789760756.066067:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25206784 chunk: [25206784, 25210880) 25210880, 4096 00020000:00200000:1.0:1789760756.066072:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.066092:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6154, 6154] 00000008:00010000:1.0:1789760756.066112:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.066119:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.066132:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25206784, 25210880) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.066150:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25206784=0x180a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.066219:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.066235:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25210880 00000020:00010000:1.0:1789760756.066244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.066248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.066251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.066255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.066258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000008:00010000:2.0:1789760756.066487:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:2.0:1789760756.066511:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771008:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.066909:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.066918:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.066938:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760756.067048:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 25206784<>25210880 00000100:00100000:0.0:1789760756.067121:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493771136:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760756.122776:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973473 gen 3 00000008:00010000:0.0:1789760756.122832:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760756.122841:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.122853:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.122939:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760756.122968:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493771136:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.125513:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6154, 6154] 00000080:00200000:1.0:1789760756.125541:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.125548:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.125568:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25210880 i_blocks 51207 00000020:00200000:1.0:1789760756.125574:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.125591:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.125626:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25210880, count: 4096 00000080:00200000:1.0:1789760756.125634:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25210880, bytes: 4096 00000080:00200000:1.0:1789760756.125648:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.125658:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.125660:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25210880, 25214976) 0 0 00000080:00200000:1.0:1789760756.125665:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.125683:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6155, 6155] 00020000:00200000:1.0:1789760756.125692:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25210880 chunk: [25210880, 25214976) 25214976, 4096 00020000:00200000:1.0:1789760756.125697:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.125719:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6155, 6155] 00000008:00010000:1.0:1789760756.125739:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.125745:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.125759:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25210880, 25214976) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.125774:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25210880=0x180b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.125841:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.125844:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25214976 00000020:00010000:1.0:1789760756.125853:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.125856:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.125860:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.125864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.125867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.125878:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.125885:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.125910:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760756.126025:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 25210880<>25214976 00000100:00100000:0.0:1789760756.126120:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493771264:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760756.199764:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973474 gen 3 00000008:00010000:0.0:1789760756.199820:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760756.199829:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.199840:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.199854:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760756.199884:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493771264:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.200072:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6155, 6155] 00000080:00200000:1.0:1789760756.200091:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.200096:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.200114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25214976 i_blocks 51207 00000020:00200000:1.0:1789760756.200119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.200130:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.200169:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25214976, count: 4096 00000080:00200000:1.0:1789760756.200179:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25214976, bytes: 4096 00000080:00200000:1.0:1789760756.200193:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.200203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.200205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25214976, 25219072) 0 0 00000080:00200000:1.0:1789760756.200209:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.200239:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6156, 6156] 00020000:00200000:1.0:1789760756.200248:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25214976 chunk: [25214976, 25219072) 25219072, 4096 00020000:00200000:1.0:1789760756.200253:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.200273:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6156, 6156] 00000008:00010000:1.0:1789760756.200291:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.200297:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.200308:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25214976, 25219072) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.200322:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25214976=0x180c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.203644:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.203658:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25219072 00000020:00010000:1.0:1789760756.203676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.203681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.203685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.203689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.203692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.203782:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.203794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.203822:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760756.203949:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 25214976<>25219072 00000100:00100000:0.0:1789760756.205255:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760756.205293:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493771392:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760756.245113:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760756.245131:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.245143:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.245287:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760756.245320:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493771392:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.245471:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6156, 6156] 00000080:00200000:2.0:1789760756.245490:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.245496:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.245512:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25219072 i_blocks 51207 00000020:00200000:2.0:1789760756.245517:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.245529:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.245567:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25219072, count: 4096 00000080:00200000:2.0:1789760756.245575:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25219072, bytes: 4096 00000080:00200000:2.0:1789760756.245591:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.245601:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.245603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25219072, 25223168) 0 0 00000080:00200000:2.0:1789760756.245608:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.245625:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6157, 6157] 00020000:00200000:2.0:1789760756.245634:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25219072 chunk: [25219072, 25223168) 25223168, 4096 00020000:00200000:2.0:1789760756.245639:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.245661:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6157, 6157] 00000008:00010000:2.0:1789760756.245681:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.245687:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.245698:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25219072, 25223168) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.245795:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25219072=0x180d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.245859:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.245863:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25223168 00000020:00010000:2.0:1789760756.245872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.245876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.245879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.245882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.245885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.245893:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.245901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.245974:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760756.246078:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 25219072<>25223168 00000100:00100000:3.0:1789760756.246161:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493771520:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.325989:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973475 gen 3 00000100:00100000:3.0:1789760756.326032:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81500 x1876699493771520/t4294973476(4294973476) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760772 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760756.326088:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760756.326095:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.326107:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.326124:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760756.326145:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493771520:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.326316:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6157, 6157] 00000080:00200000:2.0:1789760756.326333:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.326339:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.326354:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25223168 i_blocks 51207 00000020:00200000:2.0:1789760756.326358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.326368:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.326430:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25223168, count: 4096 00000080:00200000:2.0:1789760756.326437:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25223168, bytes: 4096 00000080:00200000:2.0:1789760756.326445:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.326453:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.326455:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25223168, 25227264) 0 0 00000080:00200000:2.0:1789760756.326460:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.326475:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6158, 6158] 00020000:00200000:2.0:1789760756.326483:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25223168 chunk: [25223168, 25227264) 25227264, 4096 00020000:00200000:2.0:1789760756.326487:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.326513:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6158, 6158] 00000008:00010000:2.0:1789760756.326529:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.326535:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.326546:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25223168, 25227264) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.326559:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25223168=0x180e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.326609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.326612:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25227264 00000020:00010000:2.0:1789760756.326621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.326624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.326627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.326631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.326633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.326641:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.326647:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.326662:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760756.326770:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 25223168<>25227264 00000100:00100000:3.0:1789760756.327098:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771648:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.413593:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973477 gen 3 00000008:00010000:3.0:1789760756.413662:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760756.413671:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.413682:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.413713:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760756.413744:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771648:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.413898:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6158, 6158] 00000080:00200000:2.0:1789760756.413914:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.413919:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.413946:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25227264 i_blocks 51207 00000020:00200000:2.0:1789760756.413950:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.413961:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.413995:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25227264, count: 4096 00000080:00200000:2.0:1789760756.414002:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25227264, bytes: 4096 00000080:00200000:2.0:1789760756.414011:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.414020:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.414022:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25227264, 25231360) 0 0 00000080:00200000:2.0:1789760756.414028:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.414044:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6159, 6159] 00020000:00200000:2.0:1789760756.414054:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25227264 chunk: [25227264, 25231360) 25231360, 4096 00020000:00200000:2.0:1789760756.414058:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.414078:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6159, 6159] 00000008:00010000:2.0:1789760756.414099:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.414106:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.414119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25227264, 25231360) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.414134:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25227264=0x180f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.414192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.414194:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25231360 00000020:00010000:2.0:1789760756.414203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.414207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.414210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.414214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.414216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.414234:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.414241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.414259:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760756.414365:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 25227264<>25231360 00000100:00100000:3.0:1789760756.414980:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493771776:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.467489:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973478 gen 3 00000008:00010000:3.0:1789760756.467546:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760756.467555:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.467568:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.467589:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760756.467621:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493771776:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.467840:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6159, 6159] 00000080:00200000:2.0:1789760756.467857:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.467863:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.467880:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25231360 i_blocks 51207 00000020:00200000:2.0:1789760756.467888:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.467900:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.467934:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25231360, count: 4096 00000080:00200000:2.0:1789760756.467941:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25231360, bytes: 4096 00000080:00200000:2.0:1789760756.467952:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.467961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.467963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25231360, 25235456) 0 0 00000080:00200000:2.0:1789760756.467967:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.467982:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6160, 6160] 00020000:00200000:2.0:1789760756.467991:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25231360 chunk: [25231360, 25235456) 25235456, 4096 00020000:00200000:2.0:1789760756.467995:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.468013:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6160, 6160] 00000008:00010000:2.0:1789760756.468030:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.468036:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.468046:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25231360, 25235456) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.468060:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25231360=0x1810000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.468117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.468120:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25235456 00000020:00010000:2.0:1789760756.468128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.468132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.468135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.468139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.468142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.468150:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.468157:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.468172:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760756.468773:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 25231360<>25235456 00000100:00100000:3.0:1789760756.468846:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.508587:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973479 gen 3 00000008:00010000:3.0:1789760756.508640:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760756.508649:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.508662:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.508787:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760756.508817:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493771904:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.510178:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6160, 6160] 00000080:00200000:1.0:1789760756.510201:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.510208:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.510234:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25235456 i_blocks 51207 00000020:00200000:1.0:1789760756.510239:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.510256:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.510294:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25235456, count: 4096 00000080:00200000:1.0:1789760756.510303:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25235456, bytes: 4096 00000080:00200000:1.0:1789760756.510314:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.510323:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.510325:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25235456, 25239552) 0 0 00000080:00200000:1.0:1789760756.510330:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.510346:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6161, 6161] 00020000:00200000:1.0:1789760756.510355:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25235456 chunk: [25235456, 25239552) 25239552, 4096 00020000:00200000:1.0:1789760756.510360:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.510381:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6161, 6161] 00000008:00010000:1.0:1789760756.510575:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.510582:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.510596:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25235456, 25239552) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.510611:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25235456=0x1811000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.510678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.510681:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25239552 00000020:00010000:1.0:1789760756.510690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.510693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.510696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.510699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.510702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.510713:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.510720:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.510746:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760756.510855:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 25235456<>25239552 00000100:00100000:1.0:1789760756.510915:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760756.510940:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760756.595039:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973480 gen 3 00000008:00010000:1.0:1789760756.595094:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760756.595105:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.595116:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.595139:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760756.595172:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772032:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760756.595258:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6161, 6161] 00000080:00200000:3.0:1789760756.595341:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760756.595348:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760756.595510:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25239552 i_blocks 51207 00000020:00200000:3.0:1789760756.595516:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.595771:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.595965:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25239552, count: 4096 00000080:00200000:2.0:1789760756.595974:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25239552, bytes: 4096 00000080:00200000:2.0:1789760756.595985:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.595993:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.595995:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25239552, 25243648) 0 0 00000080:00200000:2.0:1789760756.596000:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.596015:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6162, 6162] 00020000:00200000:2.0:1789760756.596024:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25239552 chunk: [25239552, 25243648) 25243648, 4096 00020000:00200000:2.0:1789760756.596029:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.596047:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6162, 6162] 00000008:00010000:2.0:1789760756.596063:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.596068:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.596079:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25239552, 25243648) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.596092:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25239552=0x1812000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.596145:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.596148:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25243648 00000020:00010000:2.0:1789760756.596155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.596158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.596161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.596165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.596168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.596176:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.596182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.596238:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760756.596493:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 25239552<>25243648 00000100:00100000:3.0:1789760756.596566:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493772160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.650788:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973481 gen 3 00000008:00010000:3.0:1789760756.650843:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760756.650852:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.650864:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.650885:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760756.650919:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493772160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.651196:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6162, 6162] 00000080:00200000:2.0:1789760756.651214:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.651232:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.651249:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25243648 i_blocks 51207 00000020:00200000:2.0:1789760756.651252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.651262:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.651303:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25243648, count: 4096 00000080:00200000:2.0:1789760756.651310:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25243648, bytes: 4096 00000080:00200000:2.0:1789760756.651320:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.651328:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.651331:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25243648, 25247744) 0 0 00000080:00200000:2.0:1789760756.651335:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.651352:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6163, 6163] 00020000:00200000:2.0:1789760756.651360:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25243648 chunk: [25243648, 25247744) 25247744, 4096 00020000:00200000:2.0:1789760756.651371:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.651427:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6163, 6163] 00000008:00010000:2.0:1789760756.651450:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.651457:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.651470:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25243648, 25247744) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.651491:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25243648=0x1813000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.651546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.651548:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25247744 00000020:00010000:2.0:1789760756.651556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.651558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.651561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.651563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.651566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.651572:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.651578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.651591:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760756.651695:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 25243648<>25247744 00000100:00100000:3.0:1789760756.651864:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493772288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.687774:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973482 gen 3 00000008:00010000:3.0:1789760756.687835:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760756.687844:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.687856:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.687875:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760756.687905:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493772288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760756.688378:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6163, 6163] 00000080:00200000:2.0:1789760756.688426:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.688433:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.688450:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25247744 i_blocks 51207 00000020:00200000:2.0:1789760756.688454:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.688465:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.688498:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25247744, count: 4096 00000080:00200000:2.0:1789760756.688506:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25247744, bytes: 4096 00000080:00200000:2.0:1789760756.688517:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.688526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.688528:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25247744, 25251840) 0 0 00000080:00200000:2.0:1789760756.688533:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.688550:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6164, 6164] 00020000:00200000:2.0:1789760756.688559:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25247744 chunk: [25247744, 25251840) 25251840, 4096 00020000:00200000:2.0:1789760756.688563:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.688583:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6164, 6164] 00000008:00010000:2.0:1789760756.688600:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.688607:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760756.688618:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25247744, 25251840) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.688631:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25247744=0x1814000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.688689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.688692:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25251840 00000020:00010000:2.0:1789760756.688871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.688876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.688880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.688883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.688886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.688895:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.688901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.688918:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760756.689171:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 25247744<>25251840 00000100:00100000:3.0:1789760756.689257:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493772416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760756.733027:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973483 gen 3 00000008:00010000:3.0:1789760756.733077:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760756.733085:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.733098:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.733119:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760756.733148:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493772416:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760756.733252:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6164, 6164] 00000080:00200000:0.0:1789760756.733268:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.733273:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.733388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25251840 i_blocks 51207 00000020:00200000:0.0:1789760756.733416:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.733428:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.733460:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25251840, count: 4096 00000080:00200000:0.0:1789760756.733467:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25251840, bytes: 4096 00000080:00200000:0.0:1789760756.733540:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.733550:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.733552:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25251840, 25255936) 0 0 00000080:00200000:0.0:1789760756.733557:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.733572:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6165, 6165] 00020000:00200000:0.0:1789760756.733581:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25251840 chunk: [25251840, 25255936) 25255936, 4096 00020000:00200000:0.0:1789760756.733586:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.733605:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6165, 6165] 00000008:00010000:0.0:1789760756.733624:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.733629:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760756.733641:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25251840, 25255936) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.733655:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25251840=0x1815000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.733778:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.733780:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25255936 00000020:00010000:0.0:1789760756.733788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.733791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.733793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.733796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.733798:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.733804:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.733810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.733894:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760756.734063:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25251840<>25255936 00000100:00100000:0.0:1789760756.734112:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760756.734130:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493772544:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760756.779330:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973484 gen 3 00000008:00010000:0.0:1789760756.779383:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760756.779391:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.779449:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.779464:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760756.779493:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493772544:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.779877:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6165, 6165] 00000080:00200000:1.0:1789760756.779892:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.779898:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.779915:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25255936 i_blocks 51207 00000020:00200000:1.0:1789760756.779919:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.779932:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.780377:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25255936, count: 4096 00000080:00200000:1.0:1789760756.780419:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25255936, bytes: 4096 00000080:00200000:1.0:1789760756.780432:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.780441:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.780443:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25255936, 25260032) 0 0 00000080:00200000:1.0:1789760756.780448:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.780465:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6166, 6166] 00020000:00200000:1.0:1789760756.780481:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25255936 chunk: [25255936, 25260032) 25260032, 4096 00020000:00200000:1.0:1789760756.780486:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.780509:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6166, 6166] 00000008:00010000:1.0:1789760756.780528:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.780535:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.780547:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25255936, 25260032) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.780562:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25255936=0x1816000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.780932:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.780935:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25260032 00000020:00010000:1.0:1789760756.780944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.781019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.781024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.781027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.781031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.781043:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.781050:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.781066:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760756.781175:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 25255936<>25260032 00000100:00100000:1.0:1789760756.781214:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760756.781250:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760756.824170:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973485 gen 3 00000008:00010000:1.0:1789760756.824207:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760756.824213:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.824231:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.824253:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760756.824285:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772672:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760756.824434:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6166, 6166] 00000080:00200000:0.0:1789760756.824447:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.824452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.824465:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25260032 i_blocks 51207 00000020:00200000:0.0:1789760756.824469:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.824479:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.824506:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25260032, count: 4096 00000080:00200000:0.0:1789760756.824513:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25260032, bytes: 4096 00000080:00200000:0.0:1789760756.824521:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.824529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.824531:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25260032, 25264128) 0 0 00000080:00200000:0.0:1789760756.824535:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.824548:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6167, 6167] 00020000:00200000:0.0:1789760756.824555:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25260032 chunk: [25260032, 25264128) 25264128, 4096 00020000:00200000:0.0:1789760756.824559:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.824575:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6167, 6167] 00000008:00010000:0.0:1789760756.824588:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.824593:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760756.824602:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25260032, 25264128) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.824613:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25260032=0x1817000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.824657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.824660:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25264128 00000020:00010000:0.0:1789760756.824667:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.824671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.824674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.824677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.824680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.824686:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.824692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.824775:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760756.824855:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25260032<>25264128 00000100:00100000:0.0:1789760756.824894:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760756.824908:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493772800:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760756.861416:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973486 gen 3 00000008:00010000:0.0:1789760756.861457:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760756.861464:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.861472:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.861494:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760756.861520:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493772800:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760756.861532:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6167, 6167] 00000080:00200000:1.0:1789760756.861548:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760756.861553:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760756.861565:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25264128 i_blocks 51207 00000020:00200000:1.0:1789760756.861569:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760756.861580:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760756.861608:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25264128, count: 4096 00000080:00200000:1.0:1789760756.861615:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25264128, bytes: 4096 00000080:00200000:1.0:1789760756.861623:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760756.861630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760756.861632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25264128, 25268224) 0 0 00000080:00200000:1.0:1789760756.861637:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760756.861650:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6168, 6168] 00020000:00200000:1.0:1789760756.861656:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25264128 chunk: [25264128, 25268224) 25268224, 4096 00020000:00200000:1.0:1789760756.861660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760756.861675:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6168, 6168] 00000008:00010000:1.0:1789760756.861689:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760756.861694:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760756.861714:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25264128, 25268224) append 0 append lockpos 0 00000080:00200000:1.0:1789760756.861725:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25264128=0x1818000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760756.861768:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760756.861770:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25268224 00000020:00010000:1.0:1789760756.861776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760756.861779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.861782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760756.861785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760756.861788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760756.861796:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760756.861801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760756.861815:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760756.861892:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 25264128<>25268224 00000100:00100000:1.0:1789760756.861929:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760756.861942:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760756.890018:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973487 gen 3 00000008:00010000:1.0:1789760756.890055:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760756.890061:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.890075:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.890193:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760756.890237:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493772928:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760756.890641:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6168, 6168] 00000080:00200000:0.0:1789760756.890655:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.890660:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.890682:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25268224 i_blocks 51207 00000020:00200000:0.0:1789760756.890686:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.890695:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.890723:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25268224, count: 4096 00000080:00200000:0.0:1789760756.890729:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25268224, bytes: 4096 00000080:00200000:0.0:1789760756.890739:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.890746:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.890749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25268224, 25272320) 0 0 00000080:00200000:0.0:1789760756.890753:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.890766:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6169, 6169] 00020000:00200000:0.0:1789760756.890774:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25268224 chunk: [25268224, 25272320) 25272320, 4096 00020000:00200000:0.0:1789760756.890778:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.890791:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6169, 6169] 00000008:00010000:0.0:1789760756.890805:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.890810:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760756.890818:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25268224, 25272320) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.890829:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25268224=0x1819000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.890873:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.890876:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25272320 00000020:00010000:0.0:1789760756.890883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.890886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.890889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.890892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.890895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.890902:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.890907:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.890924:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760756.891001:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25268224<>25272320 00000100:00100000:1.0:1789760756.891208:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773056:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760756.930314:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973488 gen 3 00000008:00010000:1.0:1789760756.930352:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760756.930358:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.930366:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.930386:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760756.930442:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773056:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760756.930452:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6169, 6169] 00000080:00200000:0.0:1789760756.930466:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.930472:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.930484:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25272320 i_blocks 51207 00000020:00200000:0.0:1789760756.930488:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.930497:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.930522:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25272320, count: 4096 00000080:00200000:0.0:1789760756.930527:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25272320, bytes: 4096 00000080:00200000:0.0:1789760756.930535:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.930542:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.930544:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25272320, 25276416) 0 0 00000080:00200000:0.0:1789760756.930548:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.930559:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6170, 6170] 00020000:00200000:0.0:1789760756.930566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25272320 chunk: [25272320, 25276416) 25276416, 4096 00020000:00200000:0.0:1789760756.930569:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.930584:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6170, 6170] 00000008:00010000:0.0:1789760756.930599:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.930603:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760756.930611:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25272320, 25276416) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.930621:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25272320=0x181a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.930665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.930668:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25276416 00000020:00010000:0.0:1789760756.930674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.930677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.930680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.930683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.930685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.930691:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.930697:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.930710:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760756.930783:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 25272320<>25276416 00000100:00100000:1.0:1789760756.930897:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773184:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760756.975487:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760756.994742:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973489 gen 3 00000008:00010000:1.0:1789760756.994779:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760756.994786:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.994795:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.994818:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760756.994844:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773184:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760756.995176:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6170, 6170] 00000080:00200000:0.0:1789760756.995188:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.995193:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.995206:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25276416 i_blocks 51207 00000020:00200000:0.0:1789760756.995209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.995228:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.995255:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25276416, count: 4096 00000080:00200000:0.0:1789760756.995261:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25276416, bytes: 4096 00000080:00200000:0.0:1789760756.995723:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.995732:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.995734:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25276416, 25280512) 0 0 00000080:00200000:0.0:1789760756.995739:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.995752:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6171, 6171] 00020000:00200000:0.0:1789760756.995759:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25276416 chunk: [25276416, 25280512) 25280512, 4096 00020000:00200000:0.0:1789760756.995762:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.995778:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6171, 6171] 00000008:00010000:0.0:1789760756.995792:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.995797:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760756.995806:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25276416, 25280512) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.995817:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25276416=0x181b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.995864:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.995867:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25280512 00000020:00010000:0.0:1789760756.995873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.995876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.995879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.995883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.995885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.995891:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.995896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.995915:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760756.996000:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 25276416<>25280512 00000100:00100000:1.0:1789760756.996069:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760757.047814:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973490 gen 3 00000008:00010000:1.0:1789760757.047849:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760757.047855:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760757.048107:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760757.048181:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760757.048210:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773312:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760757.048358:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6171, 6171] 00000080:00200000:0.0:1789760757.048370:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760757.048375:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760757.048387:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25280512 i_blocks 51207 00000020:00200000:0.0:1789760757.048390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760757.048422:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760757.048449:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25280512, count: 4096 00000080:00200000:0.0:1789760757.048455:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25280512, bytes: 4096 00000080:00200000:0.0:1789760757.048465:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760757.048472:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760757.048474:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25280512, 25284608) 0 0 00000080:00200000:0.0:1789760757.048478:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760757.048490:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6172, 6172] 00020000:00200000:0.0:1789760757.048497:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25280512 chunk: [25280512, 25284608) 25284608, 4096 00020000:00200000:0.0:1789760757.048501:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760757.048515:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6172, 6172] 00000008:00010000:0.0:1789760757.048529:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760757.048534:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760757.048544:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25280512, 25284608) append 0 append lockpos 0 00000080:00200000:0.0:1789760757.048557:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25280512=0x181c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760757.048602:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760757.048605:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25284608 00000020:00010000:0.0:1789760757.048612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760757.048615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.048618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.048622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760757.048624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760757.048631:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760757.048636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760757.048651:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760757.051601:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 25280512<>25284608 00000100:00100000:1.0:1789760757.051671:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760757.054027:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773440:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760757.118877:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760757.118888:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760757.118898:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760757.118922:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760757.118952:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773440:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760757.119026:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6172, 6172] 00000080:00200000:2.0:1789760757.119041:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.119047:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.119062:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25284608 i_blocks 51207 00000020:00200000:2.0:1789760757.119065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.119074:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.119104:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25284608, count: 4096 00000080:00200000:2.0:1789760757.119111:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25284608, bytes: 4096 00000080:00200000:2.0:1789760757.119121:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.119128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.119131:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25284608, 25288704) 0 0 00000080:00200000:2.0:1789760757.119135:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.119149:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6173, 6173] 00020000:00200000:2.0:1789760757.119156:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25284608 chunk: [25284608, 25288704) 25288704, 4096 00020000:00200000:2.0:1789760757.119159:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.119174:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6173, 6173] 00000008:00010000:2.0:1789760757.119185:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.119189:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760757.119196:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25284608, 25288704) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.119205:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25284608=0x181d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.119262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.119265:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25288704 00000020:00010000:2.0:1789760757.119342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.119346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.119349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.119352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.119355:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.119361:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.119367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.119382:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760757.119632:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 25284608<>25288704 00000100:00100000:3.0:1789760757.119683:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760757.119810:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493773568:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760757.165014:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973492 gen 3 00000008:00010000:3.0:1789760757.165061:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760757.165067:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.165076:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.165098:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760757.165122:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493773568:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.165323:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6173, 6173] 00000080:00200000:1.0:1789760757.165336:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.165343:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.165356:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25288704 i_blocks 51207 00000020:00200000:1.0:1789760757.165359:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.165368:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.165425:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25288704, count: 4096 00000080:00200000:1.0:1789760757.165431:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25288704, bytes: 4096 00000080:00200000:1.0:1789760757.165439:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.165446:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.165448:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25288704, 25292800) 0 0 00000080:00200000:1.0:1789760757.165451:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.165464:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6174, 6174] 00020000:00200000:1.0:1789760757.165471:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25288704 chunk: [25288704, 25292800) 25292800, 4096 00020000:00200000:1.0:1789760757.165475:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.165489:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6174, 6174] 00000008:00010000:1.0:1789760757.165504:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.165569:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.165581:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25288704, 25292800) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.165592:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25288704=0x181e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.165636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.165640:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25292800 00000020:00010000:1.0:1789760757.165646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.165649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.165652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.165655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.165658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.165666:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.165672:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.165686:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760757.166210:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 25288704<>25292800 00000100:00100000:0.0:1789760757.166299:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773696:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760757.200683:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760757.200694:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.200713:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.200826:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760757.200857:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493773696:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.204174:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6174, 6174] 00000080:00200000:1.0:1789760757.204197:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.204204:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.204226:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25292800 i_blocks 51207 00000020:00200000:1.0:1789760757.204231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.204245:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.204381:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25292800, count: 4096 00000080:00200000:1.0:1789760757.204390:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25292800, bytes: 4096 00000080:00200000:1.0:1789760757.204421:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.204429:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.204431:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25292800, 25296896) 0 0 00000080:00200000:1.0:1789760757.204435:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.204449:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6175, 6175] 00020000:00200000:1.0:1789760757.204457:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25292800 chunk: [25292800, 25296896) 25296896, 4096 00020000:00200000:1.0:1789760757.204461:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.204482:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6175, 6175] 00000008:00010000:1.0:1789760757.204497:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.204502:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.204581:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25292800, 25296896) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.204594:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25292800=0x181f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.204654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.204656:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25296896 00000020:00010000:1.0:1789760757.204663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.204666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.204669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.204672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.204675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.204684:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.204690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.204719:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760757.204814:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 25292800<>25296896 00000100:00100000:0.0:1789760757.206487:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773824:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.250914:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973494 gen 3 00000008:00010000:0.0:1789760757.250960:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760757.250966:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.250973:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.250992:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760757.251018:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773824:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.251050:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6175, 6175] 00000080:00200000:1.0:1789760757.251064:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.251070:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.251084:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25296896 i_blocks 51207 00000020:00200000:1.0:1789760757.251087:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.251096:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.251253:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25296896, count: 4096 00000080:00200000:1.0:1789760757.251261:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25296896, bytes: 4096 00000080:00200000:1.0:1789760757.251466:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.251475:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.251477:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25296896, 25300992) 0 0 00000080:00200000:1.0:1789760757.251481:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.251495:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6176, 6176] 00020000:00200000:1.0:1789760757.251503:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25296896 chunk: [25296896, 25300992) 25300992, 4096 00020000:00200000:1.0:1789760757.251507:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.251523:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6176, 6176] 00000008:00010000:1.0:1789760757.251538:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.251543:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.251551:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25296896, 25300992) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.251563:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25296896=0x1820000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.251606:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.251609:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25300992 00000020:00010000:1.0:1789760757.251616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.251619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.251622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.251625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.251629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.251638:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.251643:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.251662:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760757.251745:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 25296896<>25300992 00000100:00100000:0.0:1789760757.251895:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.251912:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760757.293668:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493774208:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.293757:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760757.293765:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493774080:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.293794:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493774336:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.305913:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493774208:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.305948:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493774080:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.305961:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493774336:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760757.310582:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973495 gen 3 00000008:00010000:0.0:1789760757.313277:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760757.313289:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.313299:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.313326:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:0.0:1789760757.313354:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493773952:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.313910:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6176, 6176] 00000080:00200000:1.0:1789760757.313923:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.313929:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.313942:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25300992 i_blocks 51207 00000020:00200000:1.0:1789760757.313946:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.313955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.313981:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25300992, count: 4096 00000080:00200000:1.0:1789760757.313986:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25300992, bytes: 4096 00000080:00200000:1.0:1789760757.313999:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.314006:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.314008:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25300992, 25305088) 0 0 00000080:00200000:1.0:1789760757.314012:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.314024:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6177, 6177] 00020000:00200000:1.0:1789760757.314030:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25300992 chunk: [25300992, 25305088) 25305088, 4096 00020000:00200000:1.0:1789760757.314034:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.314049:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6177, 6177] 00000008:00010000:1.0:1789760757.314062:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.314066:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.314075:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25300992, 25305088) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.314086:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25300992=0x1821000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.314127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.314130:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25305088 00000020:00010000:1.0:1789760757.314136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.314140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.314143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.314146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.314149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.314157:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.314162:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.314178:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760757.314272:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 25300992<>25305088 00000100:00100000:0.0:1789760757.315376:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774464:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.364060:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973496 gen 3 00000008:00010000:0.0:1789760757.364100:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760757.364107:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.364116:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.364147:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760757.364172:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774464:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.364355:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6177, 6177] 00000080:00200000:1.0:1789760757.364369:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.364373:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.364385:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25305088 i_blocks 51207 00000020:00200000:1.0:1789760757.364388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.364442:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.364470:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25305088, count: 4096 00000080:00200000:1.0:1789760757.364477:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25305088, bytes: 4096 00000080:00200000:1.0:1789760757.364485:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.364492:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.364494:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25305088, 25309184) 0 0 00000080:00200000:1.0:1789760757.364498:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.364511:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6178, 6178] 00020000:00200000:1.0:1789760757.364517:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25305088 chunk: [25305088, 25309184) 25309184, 4096 00020000:00200000:1.0:1789760757.364521:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.364536:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6178, 6178] 00000008:00010000:1.0:1789760757.364551:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.364556:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.364565:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25305088, 25309184) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.364576:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25305088=0x1822000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.364623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.364626:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25309184 00000020:00010000:1.0:1789760757.364632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.364636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.364638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.364642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.364645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.364654:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.364660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.364675:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760757.364753:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 25305088<>25309184 00000100:00100000:0.0:1789760757.365210:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.365237:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774592:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.426343:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973497 gen 3 00000008:00010000:0.0:1789760757.426381:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760757.426388:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.426434:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.426454:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760757.426479:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774592:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.426585:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6178, 6178] 00000080:00200000:1.0:1789760757.426599:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.426606:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.426617:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25309184 i_blocks 51207 00000020:00200000:1.0:1789760757.426621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.426630:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.426658:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25309184, count: 4096 00000080:00200000:1.0:1789760757.426664:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25309184, bytes: 4096 00000080:00200000:1.0:1789760757.426671:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.426679:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.426681:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25309184, 25313280) 0 0 00000080:00200000:1.0:1789760757.426684:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.426696:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6179, 6179] 00020000:00200000:1.0:1789760757.426703:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25309184 chunk: [25309184, 25313280) 25313280, 4096 00020000:00200000:1.0:1789760757.426707:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.426721:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6179, 6179] 00000008:00010000:1.0:1789760757.426735:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.426739:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.426748:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25309184, 25313280) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.426758:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25309184=0x1823000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.426800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.426802:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25313280 00000020:00010000:1.0:1789760757.426809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.426812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.426815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.426818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.426821:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.426828:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.426835:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.426850:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760757.426928:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 25309184<>25313280 00000100:00100000:0.0:1789760757.427031:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774720:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.475587:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973498 gen 3 00000008:00010000:0.0:1789760757.475625:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760757.475632:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.475739:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.475800:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:0.0:1789760757.475906:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774720:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.478830:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6179, 6179] 00000080:00200000:1.0:1789760757.478848:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.478854:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.478868:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25313280 i_blocks 51207 00000020:00200000:1.0:1789760757.478872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.478885:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.478911:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25313280, count: 4096 00000080:00200000:1.0:1789760757.478917:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25313280, bytes: 4096 00000080:00200000:1.0:1789760757.478928:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.478935:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.478937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25313280, 25317376) 0 0 00000080:00200000:1.0:1789760757.478940:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.478953:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6180, 6180] 00020000:00200000:1.0:1789760757.478960:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25313280 chunk: [25313280, 25317376) 25317376, 4096 00020000:00200000:1.0:1789760757.478963:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.478979:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6180, 6180] 00000008:00010000:1.0:1789760757.478993:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.478998:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.479008:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25313280, 25317376) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.479019:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25313280=0x1824000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.479076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.479079:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25317376 00000020:00010000:1.0:1789760757.479086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.479089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.479092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.479095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.479098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.479106:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.479113:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.479139:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760757.479233:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 25313280<>25317376 00000100:00100000:0.0:1789760757.479314:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493774848:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760757.520358:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760757.520369:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.520377:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.520480:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760757.520506:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493774848:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.523924:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6180, 6180] 00000080:00200000:1.0:1789760757.523944:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.523951:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.523965:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25317376 i_blocks 51207 00000020:00200000:1.0:1789760757.523970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.523983:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.524010:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25317376, count: 4096 00000080:00200000:1.0:1789760757.524018:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25317376, bytes: 4096 00000080:00200000:1.0:1789760757.524030:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.524037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.524039:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25317376, 25321472) 0 0 00000080:00200000:1.0:1789760757.524043:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.524056:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6181, 6181] 00020000:00200000:1.0:1789760757.524063:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25317376 chunk: [25317376, 25321472) 25321472, 4096 00020000:00200000:1.0:1789760757.524067:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.524084:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6181, 6181] 00000008:00010000:1.0:1789760757.524097:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.524102:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.524110:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25317376, 25321472) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.524120:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25317376=0x1825000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.524179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.524182:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25321472 00000020:00010000:1.0:1789760757.524189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.524192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.524195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.524198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.524201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.524223:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.524230:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.524256:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760757.524340:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 25317376<>25321472 00000100:00100000:0.0:1789760757.524440:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774976:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.574509:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973500 gen 3 00000008:00010000:0.0:1789760757.574558:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760757.574564:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.574572:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.574595:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760757.574619:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493774976:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.575011:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6181, 6181] 00000080:00200000:1.0:1789760757.575024:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.575029:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.575041:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25321472 i_blocks 51207 00000020:00200000:1.0:1789760757.575045:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.575054:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.575080:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25321472, count: 4096 00000080:00200000:1.0:1789760757.575085:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25321472, bytes: 4096 00000080:00200000:1.0:1789760757.575093:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.575101:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.575103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25321472, 25325568) 0 0 00000080:00200000:1.0:1789760757.575107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.575119:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6182, 6182] 00020000:00200000:1.0:1789760757.575126:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25321472 chunk: [25321472, 25325568) 25325568, 4096 00020000:00200000:1.0:1789760757.575130:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.575144:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6182, 6182] 00000008:00010000:1.0:1789760757.575158:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.575163:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.575172:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25321472, 25325568) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.575183:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25321472=0x1826000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.575239:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.575242:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25325568 00000020:00010000:1.0:1789760757.575248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.575252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.575254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.575258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.575260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.575269:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.575275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.575289:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760757.575372:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 25321472<>25325568 00000100:00100000:0.0:1789760757.575590:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775104:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.631358:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973501 gen 3 00000008:00010000:0.0:1789760757.631424:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760757.631432:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.631441:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.631460:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760757.631484:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775104:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.631800:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6182, 6182] 00000080:00200000:1.0:1789760757.631819:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.631827:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.631839:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25325568 i_blocks 51207 00000020:00200000:1.0:1789760757.631842:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.631851:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.631875:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25325568, count: 4096 00000080:00200000:1.0:1789760757.631881:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25325568, bytes: 4096 00000080:00200000:1.0:1789760757.631888:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.631895:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.631896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25325568, 25329664) 0 0 00000080:00200000:1.0:1789760757.631900:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.631912:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6183, 6183] 00020000:00200000:1.0:1789760757.631919:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25325568 chunk: [25325568, 25329664) 25329664, 4096 00020000:00200000:1.0:1789760757.631922:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.631937:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6183, 6183] 00000008:00010000:1.0:1789760757.631949:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.631954:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.631961:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25325568, 25329664) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.631971:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25325568=0x1827000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.632010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.632012:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25329664 00000020:00010000:1.0:1789760757.632018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.632021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.632024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.632027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.632029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.632122:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.632128:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.632143:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760757.632309:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 25325568<>25329664 00000100:00100000:0.0:1789760757.632369:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775232:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.678206:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973502 gen 3 00000008:00010000:0.0:1789760757.678252:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760757.678259:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.678267:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.678287:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760757.678311:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775232:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.678372:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6183, 6183] 00000080:00200000:1.0:1789760757.678385:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.678391:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.678432:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25329664 i_blocks 51207 00000020:00200000:1.0:1789760757.678435:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.678445:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.678471:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25329664, count: 4096 00000080:00200000:1.0:1789760757.678477:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25329664, bytes: 4096 00000080:00200000:1.0:1789760757.678485:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.678492:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.678494:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25329664, 25333760) 0 0 00000080:00200000:1.0:1789760757.678498:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.678516:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6184, 6184] 00020000:00200000:1.0:1789760757.678522:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25329664 chunk: [25329664, 25333760) 25333760, 4096 00020000:00200000:1.0:1789760757.678526:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.678542:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6184, 6184] 00000008:00010000:1.0:1789760757.678556:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.678560:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.678569:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25329664, 25333760) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.678580:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25329664=0x1828000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.678621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.678624:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25333760 00000020:00010000:1.0:1789760757.678630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.678633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.678636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.678639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.678642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.678650:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.678656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.678670:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760757.678748:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 25329664<>25333760 00000100:00100000:0.0:1789760757.678818:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.678831:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775360:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.711254:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973503 gen 3 00000008:00010000:0.0:1789760757.711290:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92cc0 00000020:00200000:0.0:1789760757.711296:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.711304:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.711326:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92cc0 00000100:00100000:0.0:1789760757.711350:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775360:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.711482:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6184, 6184] 00000080:00200000:1.0:1789760757.711495:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.711501:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.711512:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25333760 i_blocks 51207 00000020:00200000:1.0:1789760757.711515:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.711525:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.711551:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25333760, count: 4096 00000080:00200000:1.0:1789760757.711566:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25333760, bytes: 4096 00000080:00200000:1.0:1789760757.711573:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.711579:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.711582:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25333760, 25337856) 0 0 00000080:00200000:1.0:1789760757.711586:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.711598:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6185, 6185] 00020000:00200000:1.0:1789760757.711605:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25333760 chunk: [25333760, 25337856) 25337856, 4096 00020000:00200000:1.0:1789760757.711608:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.711623:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6185, 6185] 00000008:00010000:1.0:1789760757.711636:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.711640:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.711649:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25333760, 25337856) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.711659:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25333760=0x1829000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.711708:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.711710:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25337856 00000020:00010000:1.0:1789760757.711717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.711720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.711723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.711726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.711729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.711737:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.711742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.711757:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760757.711832:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 25333760<>25337856 00000100:00100000:0.0:1789760757.711921:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760757.711936:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775488:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.748194:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973504 gen 3 00000008:00010000:0.0:1789760757.748244:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760757.748250:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.748258:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.748276:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760757.748303:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775488:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.748444:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6185, 6185] 00000080:00200000:1.0:1789760757.748458:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.748463:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.748476:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25337856 i_blocks 51207 00000020:00200000:1.0:1789760757.748480:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.748489:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.748518:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25337856, count: 4096 00000080:00200000:1.0:1789760757.748525:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25337856, bytes: 4096 00000080:00200000:1.0:1789760757.748533:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.748540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.748542:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25337856, 25341952) 0 0 00000080:00200000:1.0:1789760757.748546:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.748571:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6186, 6186] 00020000:00200000:1.0:1789760757.748578:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25337856 chunk: [25337856, 25341952) 25341952, 4096 00020000:00200000:1.0:1789760757.748582:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.748598:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6186, 6186] 00000008:00010000:1.0:1789760757.748612:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.748616:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.748636:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25337856, 25341952) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.748648:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25337856=0x182a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.748707:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.748710:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25341952 00000020:00010000:1.0:1789760757.748718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.748722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.748724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.748727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.748730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.748737:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.748743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.748758:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760757.748847:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 25337856<>25341952 00000100:00100000:0.0:1789760757.748903:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775616:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.816471:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973505 gen 3 00000008:00010000:0.0:1789760757.816525:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92c00 00000020:00200000:0.0:1789760757.816535:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.816546:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.816571:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92c00 00000100:00100000:0.0:1789760757.816601:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493775616:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.816727:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6186, 6186] 00000080:00200000:1.0:1789760757.816743:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.816749:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.816765:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25341952 i_blocks 51207 00000020:00200000:1.0:1789760757.816768:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.816779:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.816816:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25341952, count: 4096 00000080:00200000:1.0:1789760757.816824:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25341952, bytes: 4096 00000080:00200000:1.0:1789760757.816834:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.816844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.816846:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25341952, 25346048) 0 0 00000080:00200000:1.0:1789760757.816851:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.816868:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6187, 6187] 00020000:00200000:1.0:1789760757.816877:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25341952 chunk: [25341952, 25346048) 25346048, 4096 00020000:00200000:1.0:1789760757.816880:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.816900:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6187, 6187] 00000008:00010000:1.0:1789760757.816917:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.816923:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.816936:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25341952, 25346048) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.816949:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25341952=0x182b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.817005:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.817008:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25346048 00000020:00010000:1.0:1789760757.817016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.817020:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.817024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.817027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.817030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.817041:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.817048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.817064:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760757.817165:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 25341952<>25346048 00000100:00100000:0.0:1789760757.817379:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775744:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.865123:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973506 gen 3 00000008:00010000:0.0:1789760757.865179:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:0.0:1789760757.865187:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.865199:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.865343:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:0.0:1789760757.865379:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775744:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.865555:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6187, 6187] 00000080:00200000:1.0:1789760757.865572:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.865579:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.865597:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25346048 i_blocks 51207 00000020:00200000:1.0:1789760757.865601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.865611:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.865646:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25346048, count: 4096 00000080:00200000:1.0:1789760757.865653:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25346048, bytes: 4096 00000080:00200000:1.0:1789760757.865667:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.865675:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.865678:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25346048, 25350144) 0 0 00000080:00200000:1.0:1789760757.865683:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.866174:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6188, 6188] 00020000:00200000:1.0:1789760757.866185:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25346048 chunk: [25346048, 25350144) 25350144, 4096 00020000:00200000:1.0:1789760757.866190:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.866336:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6188, 6188] 00000008:00010000:1.0:1789760757.866358:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.866364:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.866377:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25346048, 25350144) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.866392:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25346048=0x182c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.866498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.866501:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25350144 00000020:00010000:1.0:1789760757.866509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.866513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.866516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.866520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.866523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.866535:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.866543:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.866560:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760757.866676:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 25346048<>25350144 00000100:00100000:0.0:1789760757.866942:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760757.866965:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775872:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.932919:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973507 gen 3 00000008:00010000:0.0:1789760757.932963:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760757.932971:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.932981:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.933070:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760757.933102:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493775872:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.933645:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6188, 6188] 00000080:00200000:1.0:1789760757.933662:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.933669:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.933684:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25350144 i_blocks 51207 00000020:00200000:1.0:1789760757.933688:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.933948:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.933982:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25350144, count: 4096 00000080:00200000:1.0:1789760757.933990:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25350144, bytes: 4096 00000080:00200000:1.0:1789760757.934001:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.934010:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.934012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25350144, 25354240) 0 0 00000080:00200000:1.0:1789760757.934016:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.934034:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6189, 6189] 00020000:00200000:1.0:1789760757.934042:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25350144 chunk: [25350144, 25354240) 25354240, 4096 00020000:00200000:1.0:1789760757.934047:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.934064:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6189, 6189] 00000008:00010000:1.0:1789760757.934080:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.934086:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.934096:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25350144, 25354240) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.934109:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25350144=0x182d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.934168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.934171:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25354240 00000020:00010000:1.0:1789760757.934179:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.934182:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.934192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.934196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.934199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.934362:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.934374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.934423:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760757.934672:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 25350144<>25354240 00000100:00100000:0.0:1789760757.934865:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776000:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760757.996383:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973508 gen 3 00000008:00010000:0.0:1789760757.996487:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760757.996496:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.996506:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.996530:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760757.996562:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776000:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760757.996813:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6189, 6189] 00000080:00200000:1.0:1789760757.996831:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.996847:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.996930:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25354240 i_blocks 51207 00000020:00200000:1.0:1789760757.996935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.996946:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.996983:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25354240, count: 4096 00000080:00200000:1.0:1789760757.996991:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25354240, bytes: 4096 00000080:00200000:1.0:1789760757.997004:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.997012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.997015:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25354240, 25358336) 0 0 00000080:00200000:1.0:1789760757.997020:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.997037:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6190, 6190] 00020000:00200000:1.0:1789760757.997045:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25354240 chunk: [25354240, 25358336) 25358336, 4096 00020000:00200000:1.0:1789760757.997050:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.997072:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6190, 6190] 00000008:00010000:1.0:1789760757.997092:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.997099:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760757.997113:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25354240, 25358336) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.997129:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25354240=0x182e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.997192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.997195:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25358336 00000020:00010000:1.0:1789760757.997204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00010000:00010000:3.0:1789760757.998465:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00010000:1.0:1789760757.998995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.999002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.999007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.999011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.999034:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.999046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.999074:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760758.000347:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 25354240<>25358336 00000100:00100000:0.0:1789760758.000656:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776128:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760758.065029:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760758.065045:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.065057:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.065079:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760758.065108:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776128:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760758.065204:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6190, 6190] 00000080:00200000:1.0:1789760758.065233:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.065240:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.065257:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25358336 i_blocks 51207 00000020:00200000:1.0:1789760758.065261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.065272:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.065306:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25358336, count: 4096 00000080:00200000:1.0:1789760758.065315:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25358336, bytes: 4096 00000080:00200000:1.0:1789760758.065324:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.065333:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.065335:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25358336, 25362432) 0 0 00000080:00200000:1.0:1789760758.065341:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.065357:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6191, 6191] 00020000:00200000:1.0:1789760758.065366:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25358336 chunk: [25358336, 25362432) 25362432, 4096 00020000:00200000:1.0:1789760758.065371:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.065391:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6191, 6191] 00000008:00010000:1.0:1789760758.065436:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.065443:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760758.065456:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25358336, 25362432) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.065472:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25358336=0x182f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.065530:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.065533:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25362432 00000020:00010000:1.0:1789760758.065542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.065545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.065549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.065552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.065555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.065566:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.065575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.065589:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760758.065722:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 25358336<>25362432 00000100:00100000:0.0:1789760758.065787:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776256:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760758.097384:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973509 gen 3 00000100:00100000:0.0:1789760758.097456:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016ed80 x1876699493776256/t4294973510(4294973510) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760774 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760758.097630:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760758.097708:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.097718:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.097798:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760758.097831:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776256:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760758.098147:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6191, 6191] 00000080:00200000:1.0:1789760758.098161:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.098167:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.098179:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25362432 i_blocks 51207 00000020:00200000:1.0:1789760758.098183:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.098192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.098282:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25362432, count: 4096 00000080:00200000:1.0:1789760758.098379:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25362432, bytes: 4096 00000080:00200000:1.0:1789760758.098424:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.098433:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.098435:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25362432, 25366528) 0 0 00000080:00200000:1.0:1789760758.098440:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.098454:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6192, 6192] 00020000:00200000:1.0:1789760758.098461:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25362432 chunk: [25362432, 25366528) 25366528, 4096 00020000:00200000:1.0:1789760758.098464:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.098548:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6192, 6192] 00000008:00010000:1.0:1789760758.098566:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.098570:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760758.098579:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25362432, 25366528) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.098589:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25362432=0x1830000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.098745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.098748:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25366528 00000020:00010000:1.0:1789760758.098757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.098760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.098763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.098766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.098768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.098777:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.098783:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.098801:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760758.098960:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 25362432<>25366528 00000100:00100000:0.0:1789760758.099030:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776384:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760758.159514:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973511 gen 3 00000008:00010000:0.0:1789760758.159566:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760758.159573:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.159581:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.159607:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760758.159633:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776384:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760758.159925:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6192, 6192] 00000080:00200000:1.0:1789760758.159937:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.159943:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.159955:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25366528 i_blocks 51207 00000020:00200000:1.0:1789760758.159958:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.159966:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.160004:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25366528, count: 4096 00000080:00200000:1.0:1789760758.160010:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25366528, bytes: 4096 00000080:00200000:1.0:1789760758.160020:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.160027:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.160030:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25366528, 25370624) 0 0 00000080:00200000:1.0:1789760758.160033:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.160045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6193, 6193] 00020000:00200000:1.0:1789760758.160051:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25366528 chunk: [25366528, 25370624) 25370624, 4096 00020000:00200000:1.0:1789760758.160055:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.160069:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6193, 6193] 00000008:00010000:1.0:1789760758.160081:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.160086:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760758.160095:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25366528, 25370624) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.160104:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25366528=0x1831000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.160146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.160149:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25370624 00000020:00010000:1.0:1789760758.160156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.160159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.160162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.160166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.160169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.160176:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.160181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.160196:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760758.160729:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 25366528<>25370624 00000100:00100000:1.0:1789760758.160773:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776512:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760758.236108:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973512 gen 3 00000008:00010000:1.0:1789760758.236154:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760758.236164:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.236173:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.236193:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760758.236228:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493776512:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760758.236387:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6193, 6193] 00000080:00200000:0.0:1789760758.236441:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.236447:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.236460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25370624 i_blocks 51207 00000020:00200000:0.0:1789760758.236464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.236474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.236501:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25370624, count: 4096 00000080:00200000:0.0:1789760758.236508:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25370624, bytes: 4096 00000080:00200000:0.0:1789760758.236516:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.236523:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.236526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25370624, 25374720) 0 0 00000080:00200000:0.0:1789760758.236530:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.236542:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6194, 6194] 00020000:00200000:0.0:1789760758.236549:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25370624 chunk: [25370624, 25374720) 25374720, 4096 00020000:00200000:0.0:1789760758.236553:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.236567:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6194, 6194] 00000008:00010000:0.0:1789760758.236580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.236600:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760758.236609:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25370624, 25374720) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.236619:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25370624=0x1832000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.236660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.236663:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25374720 00000020:00010000:0.0:1789760758.236670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.236673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.236677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.236680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.236683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.237030:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.237036:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.237053:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760758.237143:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 25370624<>25374720 00000100:00100000:1.0:1789760758.237607:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776640:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760758.321244:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973513 gen 3 00000008:00010000:1.0:1789760758.321294:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760758.321302:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.321312:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.321332:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760758.321361:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776640:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760758.321383:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6194, 6194] 00000080:00200000:0.0:1789760758.321433:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.321445:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.321460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25374720 i_blocks 51207 00000020:00200000:0.0:1789760758.321465:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.321477:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.321505:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25374720, count: 4096 00000080:00200000:0.0:1789760758.321513:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25374720, bytes: 4096 00000080:00200000:0.0:1789760758.321525:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.321532:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.321534:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25374720, 25378816) 0 0 00000080:00200000:0.0:1789760758.321538:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.321551:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6195, 6195] 00020000:00200000:0.0:1789760758.321569:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25374720 chunk: [25374720, 25378816) 25378816, 4096 00020000:00200000:0.0:1789760758.321573:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.321597:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6195, 6195] 00000008:00010000:0.0:1789760758.321611:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.321616:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760758.321625:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25374720, 25378816) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.321635:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25374720=0x1833000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.321684:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.321687:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25378816 00000020:00010000:0.0:1789760758.321694:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.321698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.321701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.321704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dacb78 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.321707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.321713:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.321719:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.321738:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760758.324050:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 25374720<>25378816 00000100:00100000:1.0:1789760758.324226:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760758.324262:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776768:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760758.361387:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760758.361440:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.361460:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.361480:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760758.361517:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493776768:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.361622:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6195, 6195] 00000080:00200000:3.0:1789760758.361639:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.361645:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.361660:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25378816 i_blocks 51207 00000020:00200000:3.0:1789760758.361671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.361697:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.361759:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25378816, count: 4096 00000080:00200000:3.0:1789760758.361775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25378816, bytes: 4096 00000080:00200000:3.0:1789760758.361788:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.361798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.361800:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25378816, 25382912) 0 0 00000080:00200000:3.0:1789760758.361805:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.361821:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6196, 6196] 00020000:00200000:3.0:1789760758.361829:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25378816 chunk: [25378816, 25382912) 25382912, 4096 00020000:00200000:3.0:1789760758.361834:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.361852:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6196, 6196] 00000008:00010000:3.0:1789760758.361868:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.361874:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.361886:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25378816, 25382912) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.361900:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25378816=0x1834000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.361957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.361960:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25382912 00000020:00010000:3.0:1789760758.361968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.361971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.361975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.361980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.361983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.361991:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.361997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.362012:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760758.362226:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25378816<>25382912 00000100:00100000:2.0:1789760758.362335:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493776896:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.390736:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973515 gen 3 00000008:00010000:2.0:1789760758.390805:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760758.390814:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.390823:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.390843:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760758.390871:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493776896:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.391006:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6196, 6196] 00000080:00200000:3.0:1789760758.391024:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.391030:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.391048:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25382912 i_blocks 51207 00000020:00200000:3.0:1789760758.391052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.391061:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.391107:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25382912, count: 4096 00000080:00200000:3.0:1789760758.391115:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25382912, bytes: 4096 00000080:00200000:3.0:1789760758.391125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.391134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.391137:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25382912, 25387008) 0 0 00000080:00200000:3.0:1789760758.391142:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.391158:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6197, 6197] 00020000:00200000:3.0:1789760758.391166:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25382912 chunk: [25382912, 25387008) 25387008, 4096 00020000:00200000:3.0:1789760758.391171:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.391190:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6197, 6197] 00000008:00010000:3.0:1789760758.391217:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.391224:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.391235:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25382912, 25387008) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.391249:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25382912=0x1835000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.391305:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.391307:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25387008 00000020:00010000:3.0:1789760758.391315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.391319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.391322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.391325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.391328:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.391335:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.391341:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.391356:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760758.391500:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25382912<>25387008 00000100:00100000:2.0:1789760758.391649:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760758.391665:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777024:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.439241:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973516 gen 3 00000008:00010000:2.0:1789760758.439294:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760758.439305:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.439317:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.439339:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760758.439372:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777024:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.439474:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6197, 6197] 00000080:00200000:3.0:1789760758.439492:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.439498:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.439517:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25387008 i_blocks 51207 00000020:00200000:3.0:1789760758.439522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.439532:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.439569:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25387008, count: 4096 00000080:00200000:3.0:1789760758.439577:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25387008, bytes: 4096 00000080:00200000:3.0:1789760758.439587:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.439597:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.439599:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25387008, 25391104) 0 0 00000080:00200000:3.0:1789760758.439604:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.439619:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6198, 6198] 00020000:00200000:3.0:1789760758.439627:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25387008 chunk: [25387008, 25391104) 25391104, 4096 00020000:00200000:3.0:1789760758.439632:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.439652:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6198, 6198] 00000008:00010000:3.0:1789760758.439674:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.439680:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.439703:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25387008, 25391104) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.439725:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25387008=0x1836000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.439930:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.439934:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25391104 00000020:00010000:3.0:1789760758.439944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.439948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.439952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.439955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.439958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.439967:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.439973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.439989:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760758.440090:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25387008<>25391104 00000100:00100000:2.0:1789760758.440199:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777152:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.479612:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973517 gen 3 00000008:00010000:2.0:1789760758.479666:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760758.479674:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.479685:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.479990:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760758.480022:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777152:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.480115:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6198, 6198] 00000080:00200000:3.0:1789760758.480134:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.480141:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.480159:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25391104 i_blocks 51207 00000020:00200000:3.0:1789760758.480165:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.480176:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.480220:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25391104, count: 4096 00000080:00200000:3.0:1789760758.480228:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25391104, bytes: 4096 00000080:00200000:3.0:1789760758.480242:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.480251:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.480253:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25391104, 25395200) 0 0 00000080:00200000:3.0:1789760758.480369:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.480389:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6199, 6199] 00020000:00200000:3.0:1789760758.480420:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25391104 chunk: [25391104, 25395200) 25395200, 4096 00020000:00200000:3.0:1789760758.480425:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.480448:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6199, 6199] 00000008:00010000:3.0:1789760758.480467:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.480475:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.480486:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25391104, 25395200) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.480500:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25391104=0x1837000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.480557:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.480561:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25395200 00000020:00010000:3.0:1789760758.480568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.480572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.480575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.480579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.480582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.480590:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.480596:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.480613:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760758.480975:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25391104<>25395200 00000100:00100000:2.0:1789760758.481055:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493777280:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.534123:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973518 gen 3 00000008:00010000:2.0:1789760758.534170:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760758.534178:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.534188:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.534213:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760758.534246:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493777280:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.534348:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6199, 6199] 00000080:00200000:3.0:1789760758.534366:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.534372:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.534387:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25395200 i_blocks 51207 00000020:00200000:3.0:1789760758.534424:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.534435:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.534469:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25395200, count: 4096 00000080:00200000:3.0:1789760758.534477:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25395200, bytes: 4096 00000080:00200000:3.0:1789760758.534487:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.534496:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.534498:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25395200, 25399296) 0 0 00000080:00200000:3.0:1789760758.534503:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.534519:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6200, 6200] 00020000:00200000:3.0:1789760758.534527:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25395200 chunk: [25395200, 25399296) 25399296, 4096 00020000:00200000:3.0:1789760758.534532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.534552:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6200, 6200] 00000008:00010000:3.0:1789760758.534570:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.534575:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.534587:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25395200, 25399296) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.534600:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25395200=0x1838000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.534657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.534659:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25399296 00000020:00010000:3.0:1789760758.534668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.534671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.534673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.534677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.534679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.534756:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.534763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.534780:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760758.534886:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25395200<>25399296 00000100:00100000:2.0:1789760758.534948:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777408:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.572619:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973519 gen 3 00000008:00010000:2.0:1789760758.572668:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760758.572676:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.572686:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.572706:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760758.572738:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777408:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.572751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6200, 6200] 00000080:00200000:3.0:1789760758.572770:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.572776:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.572789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25399296 i_blocks 51207 00000020:00200000:3.0:1789760758.572796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.572808:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.572835:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25399296, count: 4096 00000080:00200000:3.0:1789760758.572842:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25399296, bytes: 4096 00000080:00200000:3.0:1789760758.572849:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.572856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.572858:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25399296, 25403392) 0 0 00000080:00200000:3.0:1789760758.572862:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.572874:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6201, 6201] 00020000:00200000:3.0:1789760758.572881:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25399296 chunk: [25399296, 25403392) 25403392, 4096 00020000:00200000:3.0:1789760758.572884:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.572900:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6201, 6201] 00000008:00010000:3.0:1789760758.572914:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.572919:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.572930:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25399296, 25403392) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.572940:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25399296=0x1839000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.572986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.572988:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25403392 00000020:00010000:3.0:1789760758.572994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.572997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.573000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.573003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.573006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.573012:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.573016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.573032:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760758.573103:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25399296<>25403392 00000100:00100000:2.0:1789760758.573601:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760758.573623:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777536:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.606213:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973520 gen 3 00000008:00010000:2.0:1789760758.606336:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760758.606345:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.606356:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.606378:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000080:00200000:3.0:1789760758.606437:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6201, 6201] 00000100:00100000:2.0:1789760758.606451:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777536:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.606458:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.606466:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.606484:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25403392 i_blocks 51207 00000020:00200000:3.0:1789760758.606489:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.606500:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.606533:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25403392, count: 4096 00000080:00200000:3.0:1789760758.606541:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25403392, bytes: 4096 00000080:00200000:3.0:1789760758.606550:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.606560:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.606562:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25403392, 25407488) 0 0 00000080:00200000:3.0:1789760758.606567:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.606583:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6202, 6202] 00020000:00200000:3.0:1789760758.606592:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25403392 chunk: [25403392, 25407488) 25407488, 4096 00020000:00200000:3.0:1789760758.606596:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.606615:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6202, 6202] 00000008:00010000:3.0:1789760758.606632:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.606637:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.606648:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25403392, 25407488) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.606675:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25403392=0x183a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.606732:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.606735:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25407488 00000020:00010000:3.0:1789760758.606744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.606747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.606750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.606754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.606758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.606764:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.606785:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.606802:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760758.606896:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25403392<>25407488 00000100:00100000:2.0:1789760758.607034:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777664:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.640006:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973521 gen 3 00000008:00010000:2.0:1789760758.640058:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760758.640068:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.640079:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.640098:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760758.640129:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777664:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.640362:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6202, 6202] 00000080:00200000:3.0:1789760758.640380:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.640386:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.640496:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25407488 i_blocks 51207 00000020:00200000:3.0:1789760758.640503:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.640512:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.640548:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25407488, count: 4096 00000080:00200000:3.0:1789760758.640557:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25407488, bytes: 4096 00000080:00200000:3.0:1789760758.640569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.640578:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.640580:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25407488, 25411584) 0 0 00000080:00200000:3.0:1789760758.640584:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.640600:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6203, 6203] 00020000:00200000:3.0:1789760758.640609:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25407488 chunk: [25407488, 25411584) 25411584, 4096 00020000:00200000:3.0:1789760758.640613:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.640633:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6203, 6203] 00000008:00010000:3.0:1789760758.640652:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.640658:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.640672:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25407488, 25411584) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.640889:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25407488=0x183b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.640952:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.640955:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25411584 00000020:00010000:3.0:1789760758.640964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.640968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.640971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.640975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.640978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.640987:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.640994:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.641011:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760758.641110:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25407488<>25411584 00000100:00100000:2.0:1789760758.641181:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493777792:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.685715:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973522 gen 3 00000008:00010000:2.0:1789760758.685765:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760758.685775:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.685787:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.685871:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760758.685902:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493777792:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.686874:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6203, 6203] 00000080:00200000:3.0:1789760758.686894:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.686900:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.686918:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25411584 i_blocks 51207 00000020:00200000:3.0:1789760758.686924:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.686934:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.686968:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25411584, count: 4096 00000080:00200000:3.0:1789760758.686975:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25411584, bytes: 4096 00000080:00200000:3.0:1789760758.686989:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.686997:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.687000:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25411584, 25415680) 0 0 00000080:00200000:3.0:1789760758.687004:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.687021:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6204, 6204] 00020000:00200000:3.0:1789760758.687029:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25411584 chunk: [25411584, 25415680) 25415680, 4096 00020000:00200000:3.0:1789760758.687033:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.687053:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6204, 6204] 00000008:00010000:3.0:1789760758.687071:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.687077:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.687089:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25411584, 25415680) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.687103:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25411584=0x183c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.687162:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.687165:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25415680 00000020:00010000:3.0:1789760758.687173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.687176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.687179:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.687183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.687186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.687194:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.687210:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.687229:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760758.687335:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25411584<>25415680 00000100:00100000:2.0:1789760758.687596:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777920:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.719724:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973523 gen 3 00000008:00010000:2.0:1789760758.719777:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760758.719786:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.719798:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.719824:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000080:00200000:3.0:1789760758.719853:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6204, 6204] 00000100:00100000:2.0:1789760758.719859:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493777920:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.719874:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.719881:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.719897:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25415680 i_blocks 51207 00000020:00200000:3.0:1789760758.719902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.719914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.719956:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25415680, count: 4096 00000080:00200000:3.0:1789760758.719964:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25415680, bytes: 4096 00000080:00200000:3.0:1789760758.719975:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.719984:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.719986:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25415680, 25419776) 0 0 00000080:00200000:3.0:1789760758.719991:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.720008:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6205, 6205] 00020000:00200000:3.0:1789760758.720018:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25415680 chunk: [25415680, 25419776) 25419776, 4096 00020000:00200000:3.0:1789760758.720022:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.720044:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6205, 6205] 00000008:00010000:3.0:1789760758.720064:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.720071:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.720145:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25415680, 25419776) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.720158:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25415680=0x183d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.720222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.720225:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25419776 00000020:00010000:3.0:1789760758.720234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.720237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.720241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.720244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.720247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.720256:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.720262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.720277:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760758.720382:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25415680<>25419776 00000100:00100000:2.0:1789760758.720443:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493778048:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.761635:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973524 gen 3 00000008:00010000:2.0:1789760758.761685:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760758.761694:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.761704:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.761731:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760758.761771:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493778048:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760758.761958:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6205, 6205] 00000080:00200000:0.0:1789760758.761974:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.761980:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.761994:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25419776 i_blocks 51207 00000020:00200000:0.0:1789760758.761998:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.762007:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.762039:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25419776, count: 4096 00000080:00200000:0.0:1789760758.762047:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25419776, bytes: 4096 00000080:00200000:0.0:1789760758.762058:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.762067:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.762069:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25419776, 25423872) 0 0 00000080:00200000:0.0:1789760758.762083:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.762100:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6206, 6206] 00020000:00200000:0.0:1789760758.762108:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25419776 chunk: [25419776, 25423872) 25423872, 4096 00020000:00200000:0.0:1789760758.762112:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.762132:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6206, 6206] 00000008:00010000:0.0:1789760758.762148:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.762153:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760758.762165:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25419776, 25423872) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.762178:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25419776=0x183e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.762245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.762249:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25423872 00000020:00010000:0.0:1789760758.762258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.762261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.762265:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.762269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.762272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.762281:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.762288:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.762304:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760758.762447:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 25419776<>25423872 00000100:00100000:0.0:1789760758.762481:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.762498:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778176:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760758.808774:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973525 gen 3 00000008:00010000:0.0:1789760758.808826:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760758.808836:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.808848:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.808874:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760758.808903:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760758.809232:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6206, 6206] 00000080:00200000:2.0:1789760758.809250:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.809266:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.809282:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25423872 i_blocks 51207 00000020:00200000:2.0:1789760758.809287:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.809300:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.809334:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25423872, count: 4096 00000080:00200000:2.0:1789760758.809342:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25423872, bytes: 4096 00000080:00200000:2.0:1789760758.809352:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.809361:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.809364:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25423872, 25427968) 0 0 00000080:00200000:2.0:1789760758.809370:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.809388:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6207, 6207] 00020000:00200000:2.0:1789760758.809427:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25423872 chunk: [25423872, 25427968) 25427968, 4096 00020000:00200000:2.0:1789760758.809433:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.809453:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6207, 6207] 00000008:00010000:2.0:1789760758.809472:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.809477:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760758.809490:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25423872, 25427968) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.809504:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25423872=0x183f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.809562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.809565:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25427968 00000020:00010000:2.0:1789760758.809574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.809577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.809580:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.809584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.809587:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.809595:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.809602:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.809617:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760758.809735:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 25423872<>25427968 00000100:00100000:2.0:1789760758.809779:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.809796:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493778304:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.848864:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973526 gen 3 00000008:00010000:2.0:1789760758.848914:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760758.848924:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.848935:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.849021:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760758.849054:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493778304:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760758.849195:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6207, 6207] 00000080:00200000:0.0:1789760758.849220:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.849227:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.849241:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25427968 i_blocks 51207 00000020:00200000:0.0:1789760758.849245:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.849256:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.849288:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25427968, count: 4096 00000080:00200000:0.0:1789760758.849295:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25427968, bytes: 4096 00000080:00200000:0.0:1789760758.849305:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.849315:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.849317:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25427968, 25432064) 0 0 00000080:00200000:0.0:1789760758.849321:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.849338:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6208, 6208] 00020000:00200000:0.0:1789760758.849348:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25427968 chunk: [25427968, 25432064) 25432064, 4096 00020000:00200000:0.0:1789760758.849353:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.849372:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6208, 6208] 00000008:00010000:0.0:1789760758.849391:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.849424:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760758.849437:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25427968, 25432064) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.849453:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25427968=0x1840000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.849508:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.849512:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25432064 00000020:00010000:0.0:1789760758.849519:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.849523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.849526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.849530:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.849533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.849541:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.849548:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.849564:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760758.849665:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 25427968<>25432064 00000100:00100000:1.0:1789760758.849866:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778432:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760758.898864:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760758.898881:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.898892:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.898973:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760758.899004:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778432:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760758.899256:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6208, 6208] 00000080:00200000:0.0:1789760758.899274:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.899682:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.899701:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25432064 i_blocks 51207 00000020:00200000:0.0:1789760758.899706:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.899717:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.899750:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25432064, count: 4096 00000080:00200000:0.0:1789760758.899757:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25432064, bytes: 4096 00000080:00200000:0.0:1789760758.899769:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.899778:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.899780:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25432064, 25436160) 0 0 00000080:00200000:0.0:1789760758.899785:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.899801:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6209, 6209] 00020000:00200000:0.0:1789760758.899810:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25432064 chunk: [25432064, 25436160) 25436160, 4096 00020000:00200000:0.0:1789760758.899816:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.899834:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6209, 6209] 00000008:00010000:0.0:1789760758.899851:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.899857:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760758.899868:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25432064, 25436160) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.899881:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25432064=0x1841000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.899933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.899936:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25436160 00000020:00010000:0.0:1789760758.899943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.899946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.899949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.899953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.899956:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.899963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.899970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.899986:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760758.900099:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 25432064<>25436160 00000100:00100000:1.0:1789760758.900216:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493778560:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760758.940034:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973528 gen 3 00000008:00010000:1.0:1789760758.940109:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760758.940118:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.940128:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.940149:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760758.940179:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493778560:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760758.940283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6209, 6209] 00000080:00200000:2.0:1789760758.940299:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.940305:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.940322:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25436160 i_blocks 51207 00000020:00200000:2.0:1789760758.940326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.940336:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.940368:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25436160, count: 4096 00000080:00200000:2.0:1789760758.940375:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25436160, bytes: 4096 00000080:00200000:2.0:1789760758.940385:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.940415:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.940417:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25436160, 25440256) 0 0 00000080:00200000:2.0:1789760758.940422:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.940438:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6210, 6210] 00020000:00200000:2.0:1789760758.940446:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25436160 chunk: [25436160, 25440256) 25440256, 4096 00020000:00200000:2.0:1789760758.940451:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.940472:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6210, 6210] 00000008:00010000:2.0:1789760758.940491:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.940497:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760758.940508:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25436160, 25440256) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.940524:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25436160=0x1842000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.940578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.940581:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25440256 00000020:00010000:2.0:1789760758.940589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.940593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.940606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.940610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.940614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.940623:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.940630:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.940644:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760758.940739:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 25436160<>25440256 00000100:00100000:3.0:1789760758.940893:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760758.940914:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493778688:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760758.970751:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973529 gen 3 00000008:00010000:2.0:1789760758.970805:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760758.970815:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760758.970826:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760758.970852:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760758.970882:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493778688:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760758.970902:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6210, 6210] 00000080:00200000:3.0:1789760758.970920:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.970927:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.970948:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25440256 i_blocks 51207 00000020:00200000:3.0:1789760758.970953:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.970965:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.971000:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25440256, count: 4096 00000080:00200000:3.0:1789760758.971008:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25440256, bytes: 4096 00000080:00200000:3.0:1789760758.971019:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.971029:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.971031:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25440256, 25444352) 0 0 00000080:00200000:3.0:1789760758.971037:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.971054:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6211, 6211] 00020000:00200000:3.0:1789760758.971062:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25440256 chunk: [25440256, 25444352) 25444352, 4096 00020000:00200000:3.0:1789760758.971067:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.971086:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6211, 6211] 00000008:00010000:3.0:1789760758.971105:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.971111:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760758.971123:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25440256, 25444352) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.971138:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25440256=0x1843000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.971197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.971213:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25444352 00000020:00010000:3.0:1789760758.971223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.971226:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.971230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.971233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.971236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.971245:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.971318:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.971338:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760758.971582:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25440256<>25444352 00000100:00100000:2.0:1789760758.971645:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493778816:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760759.004605:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760759.004622:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.004636:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.004653:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760759.004684:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493778816:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.004787:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6211, 6211] 00000080:00200000:0.0:1789760759.004805:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.004811:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.004827:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25444352 i_blocks 51207 00000020:00200000:0.0:1789760759.004832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.004841:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.004878:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25444352, count: 4096 00000080:00200000:0.0:1789760759.004885:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25444352, bytes: 4096 00000080:00200000:0.0:1789760759.004894:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.004903:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.004905:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25444352, 25448448) 0 0 00000080:00200000:0.0:1789760759.004910:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.004924:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6212, 6212] 00020000:00200000:0.0:1789760759.004933:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25444352 chunk: [25444352, 25448448) 25448448, 4096 00020000:00200000:0.0:1789760759.004939:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.004959:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6212, 6212] 00000008:00010000:0.0:1789760759.004978:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.004985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.004997:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25444352, 25448448) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.005013:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25444352=0x1844000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.005069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.005073:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25448448 00000020:00010000:0.0:1789760759.005081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.005084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.005088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.005091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.005095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.005102:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.005108:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.005122:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760759.005232:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 25444352<>25448448 00000100:00100000:1.0:1789760759.005348:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778944:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760759.020634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760759.057486:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973531 gen 3 00000008:00010000:1.0:1789760759.057553:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760759.057561:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.057571:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.057644:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760759.057686:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493778944:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.059994:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6212, 6212] 00000080:00200000:0.0:1789760759.060024:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.060033:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.060052:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25448448 i_blocks 51207 00000020:00200000:0.0:1789760759.060058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.060077:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.060114:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25448448, count: 4096 00000080:00200000:0.0:1789760759.060124:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25448448, bytes: 4096 00000080:00200000:0.0:1789760759.060138:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.060149:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.060151:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25448448, 25452544) 0 0 00000080:00200000:0.0:1789760759.060156:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.060174:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6213, 6213] 00020000:00200000:0.0:1789760759.060186:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25448448 chunk: [25448448, 25452544) 25452544, 4096 00020000:00200000:0.0:1789760759.060190:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.060226:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6213, 6213] 00000008:00010000:0.0:1789760759.060245:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.060379:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.060416:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25448448, 25452544) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.060434:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25448448=0x1845000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.060576:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.060579:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25452544 00000020:00010000:0.0:1789760759.060588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.060592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.060595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.060598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.060601:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.060609:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.060617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.060638:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760759.060901:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25448448<>25452544 00000100:00100000:1.0:1789760759.060991:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493779072:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.105116:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973532 gen 3 00000008:00010000:1.0:1789760759.105621:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760759.105632:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.105644:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.105775:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760759.105808:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493779072:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.105898:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6213, 6213] 00000080:00200000:3.0:1789760759.105916:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.105922:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.105938:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25452544 i_blocks 51207 00000020:00200000:3.0:1789760759.105942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.105952:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.105984:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25452544, count: 4096 00000080:00200000:3.0:1789760759.105991:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25452544, bytes: 4096 00000080:00200000:3.0:1789760759.106003:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.106012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.106015:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25452544, 25456640) 0 0 00000080:00200000:3.0:1789760759.106019:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.106126:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6214, 6214] 00020000:00200000:3.0:1789760759.106256:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25452544 chunk: [25452544, 25456640) 25456640, 4096 00020000:00200000:3.0:1789760759.106262:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.106285:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6214, 6214] 00000008:00010000:3.0:1789760759.106372:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.106378:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.106417:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25452544, 25456640) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.106433:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25452544=0x1846000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.106493:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.106496:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25456640 00000020:00010000:3.0:1789760759.106505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.106509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.106512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.106516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.106519:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.106527:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.106534:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.106625:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760759.106799:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25452544<>25456640 00000100:00100000:2.0:1789760759.106892:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493779200:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760759.149859:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760759.149877:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.149889:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.149914:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760759.149943:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493779200:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.150381:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6214, 6214] 00000080:00200000:3.0:1789760759.150426:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.150433:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.150450:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25456640 i_blocks 51207 00000020:00200000:3.0:1789760759.150454:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.150464:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.150497:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25456640, count: 4096 00000080:00200000:3.0:1789760759.150504:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25456640, bytes: 4096 00000080:00200000:3.0:1789760759.150512:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.150520:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.150521:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25456640, 25460736) 0 0 00000080:00200000:3.0:1789760759.150525:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.150540:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6215, 6215] 00020000:00200000:3.0:1789760759.150546:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25456640 chunk: [25456640, 25460736) 25460736, 4096 00020000:00200000:3.0:1789760759.150550:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.150565:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6215, 6215] 00000008:00010000:3.0:1789760759.150579:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.150583:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.150593:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25456640, 25460736) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.150603:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25456640=0x1847000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.150647:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.150649:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25460736 00000020:00010000:3.0:1789760759.150655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.150658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.150660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.150662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.150664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.150670:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.150675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.150742:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760759.150955:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25456640<>25460736 00000100:00100000:2.0:1789760759.151084:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493779328:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.194163:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973534 gen 3 00000008:00010000:2.0:1789760759.194239:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760759.194249:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.194653:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.194682:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760759.194724:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493779328:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.195083:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6215, 6215] 00000080:00200000:0.0:1789760759.195101:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.195107:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.195128:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25460736 i_blocks 51207 00000020:00200000:0.0:1789760759.195133:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.195145:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.195178:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25460736, count: 4096 00000080:00200000:0.0:1789760759.195185:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25460736, bytes: 4096 00000080:00200000:0.0:1789760759.195197:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.195216:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.195218:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25460736, 25464832) 0 0 00000080:00200000:0.0:1789760759.195223:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.195240:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6216, 6216] 00020000:00200000:0.0:1789760759.195250:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25460736 chunk: [25460736, 25464832) 25464832, 4096 00020000:00200000:0.0:1789760759.195254:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.195285:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6216, 6216] 00000008:00010000:0.0:1789760759.195305:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.195311:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.195322:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25460736, 25464832) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.195336:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25460736=0x1848000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.195426:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.195430:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25464832 00000020:00010000:0.0:1789760759.195439:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.195443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.195446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.195450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.195452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.195461:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.195468:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.195515:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760759.195626:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 25460736<>25464832 00000100:00100000:1.0:1789760759.195805:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.227077:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973535 gen 3 00000008:00010000:1.0:1789760759.227463:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760759.227472:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.227483:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.227502:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760759.227534:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779456:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.227539:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6216, 6216] 00000080:00200000:0.0:1789760759.227560:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.227566:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.227583:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25464832 i_blocks 51207 00000020:00200000:0.0:1789760759.227587:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.227599:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.227632:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25464832, count: 4096 00000080:00200000:0.0:1789760759.227639:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25464832, bytes: 4096 00000080:00200000:0.0:1789760759.227649:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.227658:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.227660:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25464832, 25468928) 0 0 00000080:00200000:0.0:1789760759.227664:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.227682:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6217, 6217] 00020000:00200000:0.0:1789760759.227691:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25464832 chunk: [25464832, 25468928) 25468928, 4096 00020000:00200000:0.0:1789760759.227696:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.227714:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6217, 6217] 00000008:00010000:0.0:1789760759.227732:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.227738:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.227750:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25464832, 25468928) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.227763:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25464832=0x1849000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.227888:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.227891:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25468928 00000020:00010000:0.0:1789760759.227901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.227905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.227908:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.227911:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.227914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.227921:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.227928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.227944:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760759.228047:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 25464832<>25468928 00000100:00100000:1.0:1789760759.228600:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.228625:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779584:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.258929:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973536 gen 3 00000008:00010000:1.0:1789760759.258982:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760759.258991:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.259002:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.259023:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760759.259052:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779584:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.259587:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6217, 6217] 00000080:00200000:0.0:1789760759.259609:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.259616:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.259632:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25468928 i_blocks 51207 00000020:00200000:0.0:1789760759.259636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.259647:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.259683:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25468928, count: 4096 00000080:00200000:0.0:1789760759.259690:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25468928, bytes: 4096 00000080:00200000:0.0:1789760759.259699:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.259708:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.259710:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25468928, 25473024) 0 0 00000080:00200000:0.0:1789760759.259715:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.259731:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6218, 6218] 00020000:00200000:0.0:1789760759.259741:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25468928 chunk: [25468928, 25473024) 25473024, 4096 00020000:00200000:0.0:1789760759.259745:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.259764:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6218, 6218] 00000008:00010000:0.0:1789760759.259782:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.259788:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.259800:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25468928, 25473024) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.259813:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25468928=0x184a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.259868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.259870:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25473024 00000020:00010000:0.0:1789760759.259878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.259881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.259884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.259887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.259890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.259897:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.259903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.259917:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760759.261650:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 25468928<>25473024 00000100:00100000:0.0:1789760759.261720:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760759.261749:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493779712:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760759.323415:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760759.323431:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.323443:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.323471:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760759.323505:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493779712:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760759.323867:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6218, 6218] 00000080:00200000:1.0:1789760759.323884:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.323890:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.323905:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25473024 i_blocks 51207 00000020:00200000:1.0:1789760759.323910:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.323921:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.324231:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25473024, count: 4096 00000080:00200000:1.0:1789760759.324241:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25473024, bytes: 4096 00000080:00200000:1.0:1789760759.324256:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.324265:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.324268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25473024, 25477120) 0 0 00000080:00200000:1.0:1789760759.324273:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.324859:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6219, 6219] 00020000:00200000:1.0:1789760759.324871:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25473024 chunk: [25473024, 25477120) 25477120, 4096 00020000:00200000:1.0:1789760759.324876:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.324898:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6219, 6219] 00000008:00010000:1.0:1789760759.324919:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.324925:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760759.324950:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25473024, 25477120) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.325027:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25473024=0x184b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.325096:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.325099:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25477120 00000020:00010000:1.0:1789760759.325109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.325113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.325115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.325118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.325121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.325132:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.325298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.325323:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760759.325472:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 25473024<>25477120 00000100:00100000:1.0:1789760759.325509:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.325533:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779840:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.355791:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973538 gen 3 00000008:00010000:1.0:1789760759.355862:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760759.355870:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.355880:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.355907:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760759.355936:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493779840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760759.356579:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6219, 6219] 00000080:00200000:2.0:1789760759.356597:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.356603:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.356618:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25477120 i_blocks 51207 00000020:00200000:2.0:1789760759.356623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.356634:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.356670:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25477120, count: 4096 00000080:00200000:2.0:1789760759.356678:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25477120, bytes: 4096 00000080:00200000:2.0:1789760759.356689:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.356698:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.356700:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25477120, 25481216) 0 0 00000080:00200000:2.0:1789760759.356705:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.356722:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6220, 6220] 00020000:00200000:2.0:1789760759.356732:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25477120 chunk: [25477120, 25481216) 25481216, 4096 00020000:00200000:2.0:1789760759.356736:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.356756:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6220, 6220] 00000008:00010000:2.0:1789760759.356774:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.356862:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760759.356877:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25477120, 25481216) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.356892:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25477120=0x184c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.356947:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.356950:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25481216 00000020:00010000:2.0:1789760759.357239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.357243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.357247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.357250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.357254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.357263:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.357270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.357557:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760759.357665:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 25477120<>25481216 00000100:00100000:3.0:1789760759.357759:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493779968:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.416290:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973539 gen 3 00000008:00010000:2.0:1789760759.418548:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760759.418563:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.418577:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.418686:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760759.418730:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493779968:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760759.419481:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6220, 6220] 00000080:00200000:1.0:1789760759.419499:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.419506:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.419523:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25481216 i_blocks 51207 00000020:00200000:1.0:1789760759.419528:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.419542:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.419636:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25481216, count: 4096 00000080:00200000:1.0:1789760759.419645:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25481216, bytes: 4096 00000080:00200000:1.0:1789760759.419662:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.419672:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.421523:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25481216, 25485312) 0 0 00000080:00200000:1.0:1789760759.421535:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.421569:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6221, 6221] 00020000:00200000:1.0:1789760759.421580:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25481216 chunk: [25481216, 25485312) 25485312, 4096 00020000:00200000:1.0:1789760759.421584:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.421611:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6221, 6221] 00000008:00010000:1.0:1789760759.421635:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.421641:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760759.421656:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25481216, 25485312) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.421672:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25481216=0x184d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.421999:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.422003:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25485312 00000020:00010000:1.0:1789760759.422012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.422015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.422018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.422021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.422024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.422039:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.422047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.422073:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760759.422206:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 25481216<>25485312 00000100:00100000:0.0:1789760759.422849:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780096:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760759.463347:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973540 gen 3 00000008:00010000:0.0:1789760759.463431:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760759.463441:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.463455:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.463477:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760759.463505:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780096:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760759.463722:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6221, 6221] 00000080:00200000:1.0:1789760759.463739:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.463745:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.463762:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25485312 i_blocks 51207 00000020:00200000:1.0:1789760759.463766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.463777:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.463814:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25485312, count: 4096 00000080:00200000:1.0:1789760759.463821:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25485312, bytes: 4096 00000080:00200000:1.0:1789760759.463832:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.463841:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.463844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25485312, 25489408) 0 0 00000080:00200000:1.0:1789760759.463848:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.463865:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6222, 6222] 00020000:00200000:1.0:1789760759.463874:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25485312 chunk: [25485312, 25489408) 25489408, 4096 00020000:00200000:1.0:1789760759.463879:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.463899:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6222, 6222] 00000008:00010000:1.0:1789760759.463931:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.463937:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760759.463951:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25485312, 25489408) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.463966:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25485312=0x184e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.464026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.464029:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25489408 00000020:00010000:1.0:1789760759.464038:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.464041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.464045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.464048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.464051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.464061:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.464069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.464083:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760759.464210:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 25485312<>25489408 00000100:00100000:0.0:1789760759.464467:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.464490:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780224:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760759.510356:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973541 gen 3 00000008:00010000:0.0:1789760759.510448:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760759.510457:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.510469:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.510539:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760759.510568:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780224:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760759.510645:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6222, 6222] 00000080:00200000:1.0:1789760759.510662:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.510668:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.510730:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25489408 i_blocks 51207 00000020:00200000:1.0:1789760759.510734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.510745:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.510777:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25489408, count: 4096 00000080:00200000:1.0:1789760759.510785:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25489408, bytes: 4096 00000080:00200000:1.0:1789760759.510794:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.510803:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.510806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25489408, 25493504) 0 0 00000080:00200000:1.0:1789760759.510811:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.510869:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6223, 6223] 00020000:00200000:1.0:1789760759.510877:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25489408 chunk: [25489408, 25493504) 25493504, 4096 00020000:00200000:1.0:1789760759.510881:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.510899:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6223, 6223] 00000008:00010000:1.0:1789760759.510917:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.510923:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760759.510935:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25489408, 25493504) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.510949:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25489408=0x184f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.511007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.511010:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25493504 00000020:00010000:1.0:1789760759.511018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.511021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.511024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.511027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.511030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.511040:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.511046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.511059:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760759.511153:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 25489408<>25493504 00000100:00100000:0.0:1789760759.511220:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780352:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760759.545460:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973542 gen 3 00000008:00010000:0.0:1789760759.545514:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760759.545524:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.545535:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.545560:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760759.545590:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780352:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760759.545892:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6223, 6223] 00000080:00200000:2.0:1789760759.545909:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.545915:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.545937:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25493504 i_blocks 51207 00000020:00200000:2.0:1789760759.545941:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.545952:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.545987:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25493504, count: 4096 00000080:00200000:2.0:1789760759.545995:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25493504, bytes: 4096 00000080:00200000:2.0:1789760759.546006:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.546014:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.546017:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25493504, 25497600) 0 0 00000080:00200000:2.0:1789760759.546022:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.546039:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6224, 6224] 00020000:00200000:2.0:1789760759.546047:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25493504 chunk: [25493504, 25497600) 25497600, 4096 00020000:00200000:2.0:1789760759.546052:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.546073:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6224, 6224] 00000008:00010000:2.0:1789760759.546091:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.546097:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760759.546108:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25493504, 25497600) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.546122:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25493504=0x1850000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.546193:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.546207:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25497600 00000020:00010000:2.0:1789760759.546214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.546218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.546221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.546225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.546228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.546235:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.546243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.546264:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760759.546371:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 25493504<>25497600 00000100:00100000:3.0:1789760759.546451:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780480:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760759.596634:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760759.596650:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.596663:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.596697:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760759.596735:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780480:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.597055:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6224, 6224] 00000080:00200000:0.0:1789760759.597072:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.597079:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.597095:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25497600 i_blocks 51207 00000020:00200000:0.0:1789760759.597099:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.597111:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.597159:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25497600, count: 4096 00000080:00200000:0.0:1789760759.597166:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25497600, bytes: 4096 00000080:00200000:0.0:1789760759.597178:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.597205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.597208:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25497600, 25501696) 0 0 00000080:00200000:0.0:1789760759.597213:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.597229:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6225, 6225] 00020000:00200000:0.0:1789760759.597246:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25497600 chunk: [25497600, 25501696) 25501696, 4096 00020000:00200000:0.0:1789760759.597251:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.597271:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6225, 6225] 00000008:00010000:0.0:1789760759.597297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.597304:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.597317:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25497600, 25501696) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.597330:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25497600=0x1851000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.597433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.597437:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25501696 00000020:00010000:0.0:1789760759.597446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.597450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.597453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.597457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.597460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.597469:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.597477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.597494:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760759.597612:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 25497600<>25501696 00000100:00100000:0.0:1789760759.597646:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.597934:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780608:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760759.647806:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973544 gen 3 00000008:00010000:0.0:1789760759.647883:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760759.647892:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.647904:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.647930:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760759.647965:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493780608:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760759.648136:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6225, 6225] 00000080:00200000:2.0:1789760759.648154:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.648160:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.648176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25501696 i_blocks 51207 00000020:00200000:2.0:1789760759.648180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.648192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.648238:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25501696, count: 4096 00000080:00200000:2.0:1789760759.648245:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25501696, bytes: 4096 00000080:00200000:2.0:1789760759.648259:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.648267:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.648269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25501696, 25505792) 0 0 00000080:00200000:2.0:1789760759.648274:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.648289:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6226, 6226] 00020000:00200000:2.0:1789760759.648298:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25501696 chunk: [25501696, 25505792) 25505792, 4096 00020000:00200000:2.0:1789760759.648302:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.648321:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6226, 6226] 00000008:00010000:2.0:1789760759.648337:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.648343:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760759.648356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25501696, 25505792) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.648371:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25501696=0x1852000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.649127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.649133:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25505792 00000020:00010000:2.0:1789760759.649142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.649146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.649149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.649152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.649154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.649164:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.649171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.649190:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760759.649305:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 25501696<>25505792 00000100:00100000:2.0:1789760759.649358:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760759.649380:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780736:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.683672:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973545 gen 3 00000008:00010000:2.0:1789760759.683727:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760759.683736:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.683747:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.683770:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760759.683799:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780736:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.683910:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6226, 6226] 00000080:00200000:3.0:1789760759.683926:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.683931:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.683947:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25505792 i_blocks 51207 00000020:00200000:3.0:1789760759.683951:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.683963:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.683997:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25505792, count: 4096 00000080:00200000:3.0:1789760759.684004:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25505792, bytes: 4096 00000080:00200000:3.0:1789760759.684014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.684022:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.684024:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25505792, 25509888) 0 0 00000080:00200000:3.0:1789760759.684028:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.684045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6227, 6227] 00020000:00200000:3.0:1789760759.684053:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25505792 chunk: [25505792, 25509888) 25509888, 4096 00020000:00200000:3.0:1789760759.684058:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.684077:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6227, 6227] 00000008:00010000:3.0:1789760759.684098:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.684104:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.684117:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25505792, 25509888) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.684132:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25505792=0x1853000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.684189:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.684192:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25509888 00000020:00010000:3.0:1789760759.684211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.684215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.684218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.684222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.684225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.684233:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.684241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.684258:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760759.684359:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25505792<>25509888 00000100:00100000:2.0:1789760759.684840:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780864:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.724476:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973546 gen 3 00000008:00010000:2.0:1789760759.724530:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760759.724540:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.724551:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.724570:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760759.724600:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.724811:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6227, 6227] 00000080:00200000:3.0:1789760759.724827:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.724834:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.724850:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25509888 i_blocks 51207 00000020:00200000:3.0:1789760759.724854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.724864:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.724900:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25509888, count: 4096 00000080:00200000:3.0:1789760759.724907:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25509888, bytes: 4096 00000080:00200000:3.0:1789760759.724916:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.724925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.724927:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25509888, 25513984) 0 0 00000080:00200000:3.0:1789760759.724932:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.724949:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6228, 6228] 00020000:00200000:3.0:1789760759.724968:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25509888 chunk: [25509888, 25513984) 25513984, 4096 00020000:00200000:3.0:1789760759.724974:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.724994:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6228, 6228] 00000008:00010000:3.0:1789760759.725014:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.725020:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.725032:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25509888, 25513984) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.725047:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25509888=0x1854000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.725107:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.725110:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25513984 00000020:00010000:3.0:1789760759.725119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.725122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.725125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.725129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.725132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.725139:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.725148:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.725164:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760759.725278:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25509888<>25513984 00000100:00100000:2.0:1789760759.725431:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760759.725451:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780992:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.758130:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973547 gen 3 00000008:00010000:2.0:1789760759.758180:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760759.758189:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.758209:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.758319:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760759.758350:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493780992:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.758469:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6228, 6228] 00000080:00200000:3.0:1789760759.758490:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.758497:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.758514:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25513984 i_blocks 51207 00000020:00200000:3.0:1789760759.758518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.758527:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.758643:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25513984, count: 4096 00000080:00200000:3.0:1789760759.758652:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25513984, bytes: 4096 00000080:00200000:3.0:1789760759.758662:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.758672:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.758817:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25513984, 25518080) 0 0 00000080:00200000:3.0:1789760759.758823:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.758841:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6229, 6229] 00020000:00200000:3.0:1789760759.758850:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25513984 chunk: [25513984, 25518080) 25518080, 4096 00020000:00200000:3.0:1789760759.758855:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.758876:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6229, 6229] 00000008:00010000:3.0:1789760759.758896:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.758902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.758915:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25513984, 25518080) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.758929:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25513984=0x1855000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.759092:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.759095:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25518080 00000020:00010000:3.0:1789760759.759105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.759109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.759112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.759116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.759119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.759128:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.759135:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.759153:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760759.760183:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25513984<>25518080 00000100:00100000:2.0:1789760759.760305:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781120:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.792971:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973548 gen 3 00000008:00010000:2.0:1789760759.793024:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760759.793033:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.793045:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.793066:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760759.793095:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.793272:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6229, 6229] 00000080:00200000:3.0:1789760759.793288:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.793294:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.793311:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25518080 i_blocks 51207 00000020:00200000:3.0:1789760759.793315:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.793324:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.793359:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25518080, count: 4096 00000080:00200000:3.0:1789760759.793367:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25518080, bytes: 4096 00000080:00200000:3.0:1789760759.793376:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.793386:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.793388:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25518080, 25522176) 0 0 00000080:00200000:3.0:1789760759.793431:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.793448:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6230, 6230] 00020000:00200000:3.0:1789760759.793458:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25518080 chunk: [25518080, 25522176) 25522176, 4096 00020000:00200000:3.0:1789760759.793462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.793483:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6230, 6230] 00000008:00010000:3.0:1789760759.793502:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.793508:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.793519:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25518080, 25522176) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.793533:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25518080=0x1856000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.793591:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.793594:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25522176 00000020:00010000:3.0:1789760759.793602:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.793606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.793609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.793613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.793616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.793625:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.793632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.793649:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760759.793761:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25518080<>25522176 00000100:00100000:2.0:1789760759.794084:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760759.794106:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781248:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.841139:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973549 gen 3 00000008:00010000:2.0:1789760759.841191:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760759.841209:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.841221:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.841253:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760759.841283:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781248:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.841421:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6230, 6230] 00000080:00200000:0.0:1789760759.841437:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.841444:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.841461:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25522176 i_blocks 51207 00000020:00200000:0.0:1789760759.841465:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.841475:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.841511:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25522176, count: 4096 00000080:00200000:0.0:1789760759.841519:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25522176, bytes: 4096 00000080:00200000:0.0:1789760759.841529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.841538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.841541:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25522176, 25526272) 0 0 00000080:00200000:0.0:1789760759.841545:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.841563:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6231, 6231] 00020000:00200000:0.0:1789760759.841572:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25522176 chunk: [25522176, 25526272) 25526272, 4096 00020000:00200000:0.0:1789760759.841577:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.841596:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6231, 6231] 00000008:00010000:0.0:1789760759.841615:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.841621:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.841634:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25522176, 25526272) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.841649:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25522176=0x1857000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.841813:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.841817:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25526272 00000020:00010000:0.0:1789760759.841826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.841830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.841833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.841836:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.841839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.841846:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.841854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.841870:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760759.842048:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 25522176<>25526272 00000100:00100000:1.0:1789760759.842251:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493781376:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.882739:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973550 gen 3 00000008:00010000:1.0:1789760759.882790:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760759.882800:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.882812:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.882889:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760759.882919:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493781376:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760759.883542:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6231, 6231] 00000080:00200000:0.0:1789760759.883560:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.883567:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.883583:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25526272 i_blocks 51207 00000020:00200000:0.0:1789760759.883588:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.883599:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.883634:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25526272, count: 4096 00000080:00200000:0.0:1789760759.883641:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25526272, bytes: 4096 00000080:00200000:0.0:1789760759.883652:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.883660:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.883662:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25526272, 25530368) 0 0 00000080:00200000:0.0:1789760759.883667:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.883741:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6232, 6232] 00020000:00200000:0.0:1789760759.883749:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25526272 chunk: [25526272, 25530368) 25530368, 4096 00020000:00200000:0.0:1789760759.883754:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.883775:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6232, 6232] 00000008:00010000:0.0:1789760759.883793:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.883800:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760759.883812:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25526272, 25530368) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.884543:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25526272=0x1858000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.884613:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.884617:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25530368 00000020:00010000:0.0:1789760759.884629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.884634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.884637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.884641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.884645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.884653:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.884661:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.884756:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760759.885808:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 25526272<>25530368 00000100:00100000:1.0:1789760759.885923:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493781504:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760759.921443:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973551 gen 3 00000008:00010000:1.0:1789760759.921490:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760759.921498:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.921508:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.921566:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760759.921596:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493781504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.923739:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6232, 6232] 00000080:00200000:3.0:1789760759.923765:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.923772:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.923855:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25530368 i_blocks 51207 00000020:00200000:3.0:1789760759.923862:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.923880:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.923916:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25530368, count: 4096 00000080:00200000:3.0:1789760759.923925:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25530368, bytes: 4096 00000080:00200000:3.0:1789760759.923939:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.923949:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.923951:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25530368, 25534464) 0 0 00000080:00200000:3.0:1789760759.923956:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.923973:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6233, 6233] 00020000:00200000:3.0:1789760759.923983:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25530368 chunk: [25530368, 25534464) 25534464, 4096 00020000:00200000:3.0:1789760759.923987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.924009:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6233, 6233] 00000008:00010000:3.0:1789760759.924028:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.924034:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.924045:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25530368, 25534464) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.924058:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25530368=0x1859000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.924905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.924910:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25534464 00000020:00010000:3.0:1789760759.924921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.924925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.924928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.924932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.924935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.924945:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.924952:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.924978:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760759.925092:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25530368<>25534464 00000100:00100000:2.0:1789760759.925203:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781632:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760759.966334:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973552 gen 3 00000008:00010000:2.0:1789760759.966528:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760759.966537:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760759.966548:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760759.966569:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760759.966600:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760759.966762:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6233, 6233] 00000080:00200000:3.0:1789760759.966780:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.966785:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.966801:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25534464 i_blocks 51207 00000020:00200000:3.0:1789760759.966805:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.966815:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.966849:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25534464, count: 4096 00000080:00200000:3.0:1789760759.966856:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25534464, bytes: 4096 00000080:00200000:3.0:1789760759.966866:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.966875:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.966877:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25534464, 25538560) 0 0 00000080:00200000:3.0:1789760759.966881:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.966898:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6234, 6234] 00020000:00200000:3.0:1789760759.966906:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25534464 chunk: [25534464, 25538560) 25538560, 4096 00020000:00200000:3.0:1789760759.966910:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.966931:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6234, 6234] 00000008:00010000:3.0:1789760759.966949:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.967022:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760759.967037:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25534464, 25538560) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.967051:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25534464=0x185a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.967109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.967113:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25538560 00000020:00010000:3.0:1789760759.967121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.967125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.967128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.967131:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.967134:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.967142:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.967149:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.967166:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760759.967347:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25534464<>25538560 00000100:00100000:2.0:1789760759.967481:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760759.967500:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781760:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760760.017701:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973553 gen 3 00000008:00010000:2.0:1789760760.017756:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760760.017765:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.017777:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.017810:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760760.017844:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493781760:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.018182:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6234, 6234] 00000080:00200000:1.0:1789760760.018209:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.018215:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.018232:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25538560 i_blocks 51207 00000020:00200000:1.0:1789760760.018236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.018246:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.018291:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25538560, count: 4096 00000080:00200000:1.0:1789760760.018299:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25538560, bytes: 4096 00000080:00200000:1.0:1789760760.018313:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.018323:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.018325:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25538560, 25542656) 0 0 00000080:00200000:1.0:1789760760.018330:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.018348:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6235, 6235] 00020000:00200000:1.0:1789760760.018357:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25538560 chunk: [25538560, 25542656) 25542656, 4096 00020000:00200000:1.0:1789760760.018361:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.018382:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6235, 6235] 00000008:00010000:1.0:1789760760.018426:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.018432:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.018445:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25538560, 25542656) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.018462:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25538560=0x185b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.018523:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.018527:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25542656 00000020:00010000:1.0:1789760760.018535:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.018538:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.018541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.018545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.018548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.018559:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.018567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.018582:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760760.018686:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 25538560<>25542656 00000100:00100000:1.0:1789760760.018731:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760760.018749:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493781888:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760760.044778:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760760.061905:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973554 gen 3 00000008:00010000:1.0:1789760760.061958:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760760.061967:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.061977:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.062054:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760760.062082:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493781888:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760760.062261:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6235, 6235] 00000080:00200000:2.0:1789760760.062277:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.062283:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.062359:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25542656 i_blocks 51207 00000020:00200000:2.0:1789760760.062364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.062374:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.062436:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25542656, count: 4096 00000080:00200000:2.0:1789760760.062443:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25542656, bytes: 4096 00000080:00200000:2.0:1789760760.062452:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.062460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.062463:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25542656, 25546752) 0 0 00000080:00200000:2.0:1789760760.062536:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.062554:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6236, 6236] 00020000:00200000:2.0:1789760760.062563:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25542656 chunk: [25542656, 25546752) 25546752, 4096 00020000:00200000:2.0:1789760760.062568:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.062588:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6236, 6236] 00000008:00010000:2.0:1789760760.062607:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.062613:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760760.062690:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25542656, 25546752) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.062705:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25542656=0x185c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.062763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.062766:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25546752 00000020:00010000:2.0:1789760760.062773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.062777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.062781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.062784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.062787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.062796:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.062804:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.062820:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760760.062988:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 25542656<>25546752 00000100:00100000:3.0:1789760760.063058:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493782016:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760760.096109:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973555 gen 3 00000008:00010000:2.0:1789760760.096166:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760760.096176:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.096205:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.096233:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760760.096264:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493782016:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.096427:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6236, 6236] 00000080:00200000:1.0:1789760760.096446:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.096453:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.096469:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25546752 i_blocks 51207 00000020:00200000:1.0:1789760760.096488:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.096499:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.096534:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25546752, count: 4096 00000080:00200000:1.0:1789760760.096543:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25546752, bytes: 4096 00000080:00200000:1.0:1789760760.096555:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.096565:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.096568:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25546752, 25550848) 0 0 00000080:00200000:1.0:1789760760.096573:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.096598:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6237, 6237] 00020000:00200000:1.0:1789760760.096607:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25546752 chunk: [25546752, 25550848) 25550848, 4096 00020000:00200000:1.0:1789760760.096611:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.096631:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6237, 6237] 00000008:00010000:1.0:1789760760.096649:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.096655:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.096675:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25546752, 25550848) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.096691:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25546752=0x185d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.096739:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.096742:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25550848 00000020:00010000:1.0:1789760760.096750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.096754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.096757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.096762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.096765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.096776:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.096783:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.097212:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:2.0:1789760760.097651:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 25546752<>25550848 00000100:00100000:3.0:1789760760.097888:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760760.098112:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493782144:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760760.135611:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:2.0:1789760760.135623:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.135632:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.135716:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:2.0:1789760760.135745:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493782144:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.136519:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6237, 6237] 00000080:00200000:1.0:1789760760.136532:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.136538:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.136549:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25550848 i_blocks 51207 00000020:00200000:1.0:1789760760.136553:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.136563:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.136592:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25550848, count: 4096 00000080:00200000:1.0:1789760760.136599:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25550848, bytes: 4096 00000080:00200000:1.0:1789760760.136614:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.136621:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.136623:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25550848, 25554944) 0 0 00000080:00200000:1.0:1789760760.136627:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.136651:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6238, 6238] 00020000:00200000:1.0:1789760760.136657:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25550848 chunk: [25550848, 25554944) 25554944, 4096 00020000:00200000:1.0:1789760760.136661:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.136675:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6238, 6238] 00000008:00010000:1.0:1789760760.136690:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.136695:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.136704:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25550848, 25554944) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.136714:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25550848=0x185e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.136773:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.136776:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25554944 00000020:00010000:1.0:1789760760.136784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.136787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.136790:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.136793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.136796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.136806:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.136812:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.136831:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760760.136930:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 25550848<>25554944 00000100:00100000:0.0:1789760760.138597:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782272:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.195996:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973557 gen 3 00000008:00010000:0.0:1789760760.196073:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760760.196082:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.196091:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.196184:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760760.196234:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782272:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.196683:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6238, 6238] 00000080:00200000:1.0:1789760760.196763:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.196770:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.196785:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25554944 i_blocks 51207 00000020:00200000:1.0:1789760760.196788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.196798:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.196829:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25554944, count: 4096 00000080:00200000:1.0:1789760760.196835:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25554944, bytes: 4096 00000080:00200000:1.0:1789760760.196846:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.196854:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.196856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25554944, 25559040) 0 0 00000080:00200000:1.0:1789760760.196862:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.196877:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6239, 6239] 00020000:00200000:1.0:1789760760.196886:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25554944 chunk: [25554944, 25559040) 25559040, 4096 00020000:00200000:1.0:1789760760.196890:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.196907:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6239, 6239] 00000008:00010000:1.0:1789760760.196923:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.196929:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.196939:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25554944, 25559040) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.196952:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25554944=0x185f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.198270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.198288:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25559040 00000020:00010000:1.0:1789760760.198303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.198307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.198311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.198315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.198318:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.198332:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.198339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.198364:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760760.198494:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 25554944<>25559040 00000100:00100000:0.0:1789760760.198693:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760760.198720:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782400:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.264790:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973558 gen 3 00000008:00010000:0.0:1789760760.264836:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760760.264843:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.264851:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.264870:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760760.264896:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782400:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.264951:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6239, 6239] 00000080:00200000:1.0:1789760760.264974:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.264981:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.265009:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25559040 i_blocks 51207 00000020:00200000:1.0:1789760760.265015:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.265028:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.265064:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25559040, count: 4096 00000080:00200000:1.0:1789760760.265072:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25559040, bytes: 4096 00000080:00200000:1.0:1789760760.265084:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.265093:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.265095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25559040, 25563136) 0 0 00000080:00200000:1.0:1789760760.265100:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.265116:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6240, 6240] 00020000:00200000:1.0:1789760760.265124:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25559040 chunk: [25559040, 25563136) 25563136, 4096 00020000:00200000:1.0:1789760760.265128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.265148:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6240, 6240] 00000008:00010000:1.0:1789760760.265166:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.265171:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.265182:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25559040, 25563136) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.265205:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25559040=0x1860000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.265264:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.265267:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25563136 00000020:00010000:1.0:1789760760.265276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.265279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.265282:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.265286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.265289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.265309:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.265315:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.265333:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760760.265624:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 25559040<>25563136 00000100:00100000:0.0:1789760760.265799:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782528:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.305472:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973559 gen 3 00000008:00010000:0.0:1789760760.305618:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760760.305627:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.305638:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.305664:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760760.305696:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493782528:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.305857:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6240, 6240] 00000080:00200000:1.0:1789760760.305877:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.305883:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.305899:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25563136 i_blocks 51207 00000020:00200000:1.0:1789760760.305903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.305914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.305950:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25563136, count: 4096 00000080:00200000:1.0:1789760760.305958:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25563136, bytes: 4096 00000080:00200000:1.0:1789760760.305968:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.305977:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.305980:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25563136, 25567232) 0 0 00000080:00200000:1.0:1789760760.305985:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.306002:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6241, 6241] 00020000:00200000:1.0:1789760760.306011:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25563136 chunk: [25563136, 25567232) 25567232, 4096 00020000:00200000:1.0:1789760760.306015:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.306035:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6241, 6241] 00000008:00010000:1.0:1789760760.306054:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.306060:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.306072:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25563136, 25567232) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.306086:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25563136=0x1861000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.306140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.306144:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25567232 00000020:00010000:1.0:1789760760.306153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.306157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.306160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.306163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.306166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.306177:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.306185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.306647:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:0.0:1789760760.306776:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 25563136<>25567232 00000100:00100000:0.0:1789760760.306826:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782656:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.360552:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973560 gen 3 00000008:00010000:0.0:1789760760.360615:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:0.0:1789760760.360623:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.360635:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.360659:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:0.0:1789760760.360690:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782656:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.360847:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6241, 6241] 00000080:00200000:1.0:1789760760.360865:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.360870:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.360886:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25567232 i_blocks 51207 00000020:00200000:1.0:1789760760.360890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.360901:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.361064:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25567232, count: 4096 00000080:00200000:1.0:1789760760.361073:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25567232, bytes: 4096 00000080:00200000:1.0:1789760760.361084:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.361093:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.361095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25567232, 25571328) 0 0 00000080:00200000:1.0:1789760760.361100:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.361116:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6242, 6242] 00020000:00200000:1.0:1789760760.361125:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25567232 chunk: [25567232, 25571328) 25571328, 4096 00020000:00200000:1.0:1789760760.361129:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.361149:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6242, 6242] 00000008:00010000:1.0:1789760760.361170:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.361177:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.361200:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25567232, 25571328) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.361217:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25567232=0x1862000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.361277:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.361280:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25571328 00000020:00010000:1.0:1789760760.361290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.362384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.362412:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.362417:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.362420:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.362435:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.362445:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.362472:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760760.362592:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 25567232<>25571328 00000100:00100000:0.0:1789760760.363442:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760760.363486:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782784:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760760.460143:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92c00 00000020:00200000:0.0:1789760760.460154:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.460163:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.460185:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92c00 00000100:00100000:0.0:1789760760.460220:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782784:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.460440:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6242, 6242] 00000080:00200000:1.0:1789760760.460453:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.460459:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.460470:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25571328 i_blocks 51207 00000020:00200000:1.0:1789760760.460474:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.460482:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.460579:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25571328, count: 4096 00000080:00200000:1.0:1789760760.460587:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25571328, bytes: 4096 00000080:00200000:1.0:1789760760.460595:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.460601:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.460603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25571328, 25575424) 0 0 00000080:00200000:1.0:1789760760.460607:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.460620:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6243, 6243] 00020000:00200000:1.0:1789760760.460627:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25571328 chunk: [25571328, 25575424) 25575424, 4096 00020000:00200000:1.0:1789760760.460631:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.460646:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6243, 6243] 00000008:00010000:1.0:1789760760.460660:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.460664:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.460684:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25571328, 25575424) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.460695:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25571328=0x1863000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.460738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.460741:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25575424 00000020:00010000:1.0:1789760760.460747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.460751:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.460753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.460756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.460759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.460768:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.460774:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.460790:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760760.460878:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 25571328<>25575424 00000100:00100000:0.0:1789760760.460939:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782912:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.523987:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973562 gen 3 00000008:00010000:0.0:1789760760.524038:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760760.524045:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.524052:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.524073:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760760.524105:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493782912:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760760.524230:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6243, 6243] 00000080:00200000:1.0:1789760760.524245:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.524251:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.524264:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25575424 i_blocks 51207 00000020:00200000:1.0:1789760760.524268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.524277:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.524307:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25575424, count: 4096 00000080:00200000:1.0:1789760760.524315:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25575424, bytes: 4096 00000080:00200000:1.0:1789760760.524324:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.524332:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.524334:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25575424, 25579520) 0 0 00000080:00200000:1.0:1789760760.524339:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.524353:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6244, 6244] 00020000:00200000:1.0:1789760760.524361:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25575424 chunk: [25575424, 25579520) 25579520, 4096 00020000:00200000:1.0:1789760760.524366:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.524382:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6244, 6244] 00000008:00010000:1.0:1789760760.524433:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.524439:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760760.524450:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25575424, 25579520) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.524462:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25575424=0x1864000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.524510:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.524512:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25579520 00000020:00010000:1.0:1789760760.524520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.524523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.524526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.524529:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.524532:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.524541:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.524546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.524560:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92cc0 00000008:00100000:1.0:1789760760.524636:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 25575424<>25579520 00000100:00100000:0.0:1789760760.524761:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493783040:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.576812:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973563 gen 3 00000008:00010000:0.0:1789760760.576858:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92cc0 00000020:00200000:0.0:1789760760.576867:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.576879:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.576909:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92cc0 00000100:00100000:0.0:1789760760.576940:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493783040:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760760.577201:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6244, 6244] 00000080:00200000:2.0:1789760760.577214:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.577220:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.577232:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25579520 i_blocks 51207 00000020:00200000:2.0:1789760760.577236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.577244:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.577270:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25579520, count: 4096 00000080:00200000:2.0:1789760760.577277:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25579520, bytes: 4096 00000080:00200000:2.0:1789760760.577295:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.577302:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.577304:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25579520, 25583616) 0 0 00000080:00200000:2.0:1789760760.577308:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.577322:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6245, 6245] 00020000:00200000:2.0:1789760760.577328:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25579520 chunk: [25579520, 25583616) 25583616, 4096 00020000:00200000:2.0:1789760760.577332:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.577346:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6245, 6245] 00000008:00010000:2.0:1789760760.577358:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.577363:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760760.577371:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25579520, 25583616) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.577380:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25579520=0x1865000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.577455:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.577458:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25583616 00000020:00010000:2.0:1789760760.577466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.577469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.577472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.577475:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.577479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.577486:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.577658:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.577674:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760760.577765:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 25579520<>25583616 00000100:00100000:3.0:1789760760.577841:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783168:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760760.687562:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973564 gen 3 00000008:00010000:2.0:1789760760.687600:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760760.687606:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.687615:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.687635:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760760.687660:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783168:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760760.687748:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6245, 6245] 00000080:00200000:0.0:1789760760.687760:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.687766:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.687778:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25583616 i_blocks 51207 00000020:00200000:0.0:1789760760.687782:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.687790:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.687820:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25583616, count: 4096 00000080:00200000:0.0:1789760760.687825:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25583616, bytes: 4096 00000080:00200000:0.0:1789760760.687834:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.687842:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.687844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25583616, 25587712) 0 0 00000080:00200000:0.0:1789760760.687847:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.687859:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6246, 6246] 00020000:00200000:0.0:1789760760.687865:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25583616 chunk: [25583616, 25587712) 25587712, 4096 00020000:00200000:0.0:1789760760.687869:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.687884:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6246, 6246] 00000008:00010000:0.0:1789760760.687897:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.687902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760760.687910:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25583616, 25587712) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.687919:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25583616=0x1866000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.687962:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.687965:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25587712 00000020:00010000:0.0:1789760760.687972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.687975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.687977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.687980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.687983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.687990:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.687995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.688010:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760760.688109:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 25583616<>25587712 00000100:00100000:0.0:1789760760.688150:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760760.688161:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493783296:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760760.772477:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973565 gen 3 00000008:00010000:0.0:1789760760.772519:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760760.772526:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.772534:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.772563:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760760.772590:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493783296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760760.772849:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6246, 6246] 00000080:00200000:2.0:1789760760.772861:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.772867:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.772878:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25587712 i_blocks 51207 00000020:00200000:2.0:1789760760.772881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.772890:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.772917:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25587712, count: 4096 00000080:00200000:2.0:1789760760.772922:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25587712, bytes: 4096 00000080:00200000:2.0:1789760760.772930:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.772937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.772939:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25587712, 25591808) 0 0 00000080:00200000:2.0:1789760760.772943:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.772955:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6247, 6247] 00020000:00200000:2.0:1789760760.772962:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25587712 chunk: [25587712, 25591808) 25591808, 4096 00020000:00200000:2.0:1789760760.772965:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.772979:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6247, 6247] 00000008:00010000:2.0:1789760760.772992:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.772997:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760760.773006:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25587712, 25591808) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.773016:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25587712=0x1867000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.773060:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.773063:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25591808 00000020:00010000:2.0:1789760760.773069:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.773072:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.773075:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.773078:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.773081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.773086:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.773091:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.773105:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760760.773200:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 25587712<>25591808 00000100:00100000:2.0:1789760760.773247:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760760.773260:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783424:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760760.859183:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760760.859203:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.859214:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.859239:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760760.859266:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783424:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760760.859430:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6247, 6247] 00000080:00200000:3.0:1789760760.859442:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.859447:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.859459:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25591808 i_blocks 51207 00000020:00200000:3.0:1789760760.859462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.859470:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.859509:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25591808, count: 4096 00000080:00200000:3.0:1789760760.859516:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25591808, bytes: 4096 00000080:00200000:3.0:1789760760.859525:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.859532:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.859534:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25591808, 25595904) 0 0 00000080:00200000:3.0:1789760760.859539:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.859560:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6248, 6248] 00020000:00200000:3.0:1789760760.859566:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25591808 chunk: [25591808, 25595904) 25595904, 4096 00020000:00200000:3.0:1789760760.859570:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.859586:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6248, 6248] 00000008:00010000:3.0:1789760760.859600:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.859606:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760760.859615:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25591808, 25595904) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.859626:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25591808=0x1868000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.859669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.859672:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25595904 00000020:00010000:3.0:1789760760.859679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.859682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.859685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.859689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.859692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.859697:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.859703:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.859717:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760760.859791:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25591808<>25595904 00000100:00100000:3.0:1789760760.859829:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760760.859841:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783552:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760760.947608:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973566 gen 3 00000100:00100000:2.0:1789760760.947639:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b100 x1876699493783552/t4294973567(4294973567) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760776 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760760.947691:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760760.947697:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.947706:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.947723:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760760.947759:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783552:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760760.947929:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6248, 6248] 00000080:00200000:3.0:1789760760.947942:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.947947:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.947958:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25595904 i_blocks 51207 00000020:00200000:3.0:1789760760.947961:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.947969:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.947997:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25595904, count: 4096 00000080:00200000:3.0:1789760760.948003:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25595904, bytes: 4096 00000080:00200000:3.0:1789760760.948011:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.948018:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.948020:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25595904, 25600000) 0 0 00000080:00200000:3.0:1789760760.948024:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.948037:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6249, 6249] 00020000:00200000:3.0:1789760760.948044:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25595904 chunk: [25595904, 25600000) 25600000, 4096 00020000:00200000:3.0:1789760760.948048:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.948063:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6249, 6249] 00000008:00010000:3.0:1789760760.948076:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.948080:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760760.948088:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25595904, 25600000) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.948100:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25595904=0x1869000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.948143:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.948146:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25600000 00000020:00010000:3.0:1789760760.948152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.948155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.948158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.948161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.948164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.948170:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.948175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.949438:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760760.949532:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25595904<>25600000 00000100:00100000:2.0:1789760760.949835:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783680:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760760.982511:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973567 gen 3 00000100:00100000:2.0:1789760760.982542:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149c00 x1876699493783680/t4294973568(4294973568) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760776 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760760.982580:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760760.982586:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760760.982593:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760760.982615:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760760.982637:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783680:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760760.982882:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6249, 6249] 00000080:00200000:3.0:1789760760.982895:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.982900:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.982912:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25600000 i_blocks 51207 00000020:00200000:3.0:1789760760.982915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.982923:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.982949:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25600000, count: 4096 00000080:00200000:3.0:1789760760.982955:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25600000, bytes: 4096 00000080:00200000:3.0:1789760760.982964:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.982971:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.982973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25600000, 25604096) 0 0 00000080:00200000:3.0:1789760760.982977:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.982990:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6250, 6250] 00020000:00200000:3.0:1789760760.982997:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25600000 chunk: [25600000, 25604096) 25604096, 4096 00020000:00200000:3.0:1789760760.983001:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.983014:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6250, 6250] 00000008:00010000:3.0:1789760760.983027:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.983032:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760760.983042:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25600000, 25604096) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.983053:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25600000=0x186a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.983098:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.983101:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25604096 00000020:00010000:3.0:1789760760.983107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.983110:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.983113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.983117:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.983120:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.983126:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.983131:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.983144:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760760.983232:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25600000<>25604096 00000100:00100000:2.0:1789760760.983365:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783808:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.017064:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973569 gen 3 00000008:00010000:2.0:1789760761.017115:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760761.017122:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.017129:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.017146:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760761.017171:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493783808:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.017373:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6250, 6250] 00000080:00200000:3.0:1789760761.017386:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.017431:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.017442:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25604096 i_blocks 51207 00000020:00200000:3.0:1789760761.017445:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.017454:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.017481:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25604096, count: 4096 00000080:00200000:3.0:1789760761.017487:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25604096, bytes: 4096 00000080:00200000:3.0:1789760761.017495:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.017503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.017505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25604096, 25608192) 0 0 00000080:00200000:3.0:1789760761.017509:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.017523:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6251, 6251] 00020000:00200000:3.0:1789760761.017530:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25604096 chunk: [25604096, 25608192) 25608192, 4096 00020000:00200000:3.0:1789760761.017533:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.017547:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6251, 6251] 00000008:00010000:3.0:1789760761.017560:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.017565:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.017574:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25604096, 25608192) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.017586:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25604096=0x186b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.017632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.017635:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25608192 00000020:00010000:3.0:1789760761.017641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.017645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.017647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.017651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.017654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.017660:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.017677:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.017691:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760761.017772:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 25604096<>25608192 00000100:00100000:2.0:1789760761.017843:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783936:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760761.072771:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760761.104895:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973570 gen 3 00000008:00010000:2.0:1789760761.104932:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760761.104939:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.104948:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.104967:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760761.104992:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493783936:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.105178:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6251, 6251] 00000080:00200000:3.0:1789760761.105201:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.105206:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.105218:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25608192 i_blocks 51207 00000020:00200000:3.0:1789760761.105222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.105229:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.105264:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25608192, count: 4096 00000080:00200000:3.0:1789760761.105270:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25608192, bytes: 4096 00000080:00200000:3.0:1789760761.105277:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.105284:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.105286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25608192, 25612288) 0 0 00000080:00200000:3.0:1789760761.105290:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.105306:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6252, 6252] 00020000:00200000:3.0:1789760761.105313:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25608192 chunk: [25608192, 25612288) 25612288, 4096 00020000:00200000:3.0:1789760761.105318:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.105334:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6252, 6252] 00000008:00010000:3.0:1789760761.105348:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.105353:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.105361:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25608192, 25612288) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.105373:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25608192=0x186c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.105442:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.105445:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25612288 00000020:00010000:3.0:1789760761.105452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.105456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.105459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.105462:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.105465:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.105471:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.105477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.105492:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760761.105571:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 25608192<>25612288 00000100:00100000:2.0:1789760761.105661:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.105684:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784064:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760761.142299:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760761.142311:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.142319:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.142337:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760761.142364:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784064:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.142478:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6252, 6252] 00000080:00200000:3.0:1789760761.142489:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.142494:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.142506:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25612288 i_blocks 51207 00000020:00200000:3.0:1789760761.142509:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.142517:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.142544:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25612288, count: 4096 00000080:00200000:3.0:1789760761.142551:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25612288, bytes: 4096 00000080:00200000:3.0:1789760761.142560:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.142567:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.142569:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25612288, 25616384) 0 0 00000080:00200000:3.0:1789760761.142573:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.142587:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6253, 6253] 00020000:00200000:3.0:1789760761.142595:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25612288 chunk: [25612288, 25616384) 25616384, 4096 00020000:00200000:3.0:1789760761.142598:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.142612:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6253, 6253] 00000008:00010000:3.0:1789760761.142624:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.142629:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.142638:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25612288, 25616384) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.142650:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25612288=0x186d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.142704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.142707:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25616384 00000020:00010000:3.0:1789760761.142714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.142717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.142720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.142723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.142726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.142732:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.142738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.142753:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760761.142829:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25612288<>25616384 00000100:00100000:2.0:1789760761.142969:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784192:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.209248:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973572 gen 3 00000008:00010000:2.0:1789760761.209297:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760761.209302:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.209310:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.209327:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760761.209351:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784192:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.209513:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6253, 6253] 00000080:00200000:3.0:1789760761.209527:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.209532:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.209544:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25616384 i_blocks 51207 00000020:00200000:3.0:1789760761.209548:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.209555:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.209582:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25616384, count: 4096 00000080:00200000:3.0:1789760761.209588:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25616384, bytes: 4096 00000080:00200000:3.0:1789760761.209597:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.209603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.209605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25616384, 25620480) 0 0 00000080:00200000:3.0:1789760761.209609:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.209621:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6254, 6254] 00020000:00200000:3.0:1789760761.209628:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25616384 chunk: [25616384, 25620480) 25620480, 4096 00020000:00200000:3.0:1789760761.209631:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.209646:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6254, 6254] 00000008:00010000:3.0:1789760761.209659:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.209676:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.209686:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25616384, 25620480) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.209698:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25616384=0x186e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.209749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.209752:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25620480 00000020:00010000:3.0:1789760761.209759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.209762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.209765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.209768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.209771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.209777:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.209783:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.209798:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760761.209892:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25616384<>25620480 00000100:00100000:2.0:1789760761.210272:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.210289:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784320:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.248739:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973573 gen 3 00000008:00010000:2.0:1789760761.248777:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760761.248784:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.248791:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.248811:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760761.248847:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784320:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.248944:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6254, 6254] 00000080:00200000:3.0:1789760761.248960:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.248964:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.248976:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25620480 i_blocks 51207 00000020:00200000:3.0:1789760761.248979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.248986:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.249015:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25620480, count: 4096 00000080:00200000:3.0:1789760761.249021:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25620480, bytes: 4096 00000080:00200000:3.0:1789760761.249028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.249034:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.249036:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25620480, 25624576) 0 0 00000080:00200000:3.0:1789760761.249040:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.249052:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6255, 6255] 00020000:00200000:3.0:1789760761.249058:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25620480 chunk: [25620480, 25624576) 25624576, 4096 00020000:00200000:3.0:1789760761.249061:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.249073:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6255, 6255] 00000008:00010000:3.0:1789760761.249084:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.249089:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.249096:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25620480, 25624576) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.249106:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25620480=0x186f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.249144:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.249146:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25624576 00000020:00010000:3.0:1789760761.249152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.249155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.249157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.249160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.249162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.249167:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.249172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.249193:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760761.249265:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25620480<>25624576 00000100:00100000:2.0:1789760761.249350:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784448:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.296958:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973574 gen 3 00000008:00010000:2.0:1789760761.296999:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760761.297005:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.297013:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.297033:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760761.297058:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784448:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.297061:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6255, 6255] 00000080:00200000:3.0:1789760761.297075:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.297080:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.297092:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25624576 i_blocks 51207 00000020:00200000:3.0:1789760761.297096:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.297104:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.297133:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25624576, count: 4096 00000080:00200000:3.0:1789760761.297140:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25624576, bytes: 4096 00000080:00200000:3.0:1789760761.297149:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.297157:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.297159:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25624576, 25628672) 0 0 00000080:00200000:3.0:1789760761.297163:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.297175:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6256, 6256] 00020000:00200000:3.0:1789760761.297192:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25624576 chunk: [25624576, 25628672) 25628672, 4096 00020000:00200000:3.0:1789760761.297196:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.297212:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6256, 6256] 00000008:00010000:3.0:1789760761.297225:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.297230:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.297368:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25624576, 25628672) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.297381:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25624576=0x1870000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.297460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.297463:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25628672 00000020:00010000:3.0:1789760761.297470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.297474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.297543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.297548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.297551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.297558:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.297564:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.297578:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760761.297653:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25624576<>25628672 00000100:00100000:2.0:1789760761.297780:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493784576:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.349178:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973575 gen 3 00000008:00010000:2.0:1789760761.349227:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760761.349233:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.349257:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.349275:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760761.349300:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493784576:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.349457:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6256, 6256] 00000080:00200000:3.0:1789760761.349471:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.349476:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.349487:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25628672 i_blocks 51207 00000020:00200000:3.0:1789760761.349491:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.349499:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.349525:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25628672, count: 4096 00000080:00200000:3.0:1789760761.349535:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25628672, bytes: 4096 00000080:00200000:3.0:1789760761.349544:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.349551:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.349553:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25628672, 25632768) 0 0 00000080:00200000:3.0:1789760761.349557:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.349569:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6257, 6257] 00020000:00200000:3.0:1789760761.349577:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25628672 chunk: [25628672, 25632768) 25632768, 4096 00020000:00200000:3.0:1789760761.349580:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.349595:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6257, 6257] 00000008:00010000:3.0:1789760761.349608:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.349613:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.349622:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25628672, 25632768) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.349633:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25628672=0x1871000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.349689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.349691:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25632768 00000020:00010000:3.0:1789760761.349698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.349702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.349705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.349708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.349711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.349717:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.349723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.349737:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760761.349813:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25628672<>25632768 00000100:00100000:2.0:1789760761.350165:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784704:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.400090:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973576 gen 3 00000008:00010000:2.0:1789760761.400128:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760761.400134:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.400142:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.400162:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760761.400199:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784704:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.400576:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6257, 6257] 00000080:00200000:3.0:1789760761.400591:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.400595:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.400606:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25632768 i_blocks 51207 00000020:00200000:3.0:1789760761.400610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.400618:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.400646:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25632768, count: 4096 00000080:00200000:3.0:1789760761.400652:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25632768, bytes: 4096 00000080:00200000:3.0:1789760761.400806:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.400814:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.400824:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25632768, 25636864) 0 0 00000080:00200000:3.0:1789760761.400828:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.400841:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6258, 6258] 00020000:00200000:3.0:1789760761.400848:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25632768 chunk: [25632768, 25636864) 25636864, 4096 00020000:00200000:3.0:1789760761.400852:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.400869:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6258, 6258] 00000008:00010000:3.0:1789760761.400884:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.400889:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.400899:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25632768, 25636864) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.400911:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25632768=0x1872000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.400959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.400962:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25636864 00000020:00010000:3.0:1789760761.400969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.400972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.400975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.400978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.400980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.400986:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.400991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.401006:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760761.401087:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25632768<>25636864 00000100:00100000:2.0:1789760761.401154:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493784832:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.451030:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973577 gen 3 00000008:00010000:2.0:1789760761.451067:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760761.451073:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.451081:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.451101:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760761.451127:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493784832:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.451337:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6258, 6258] 00000080:00200000:3.0:1789760761.451350:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.451354:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.451366:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25636864 i_blocks 51207 00000020:00200000:3.0:1789760761.451370:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.451379:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.451436:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25636864, count: 4096 00000080:00200000:3.0:1789760761.451443:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25636864, bytes: 4096 00000080:00200000:3.0:1789760761.451451:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.451458:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.451460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25636864, 25640960) 0 0 00000080:00200000:3.0:1789760761.451464:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.451476:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6259, 6259] 00020000:00200000:3.0:1789760761.451482:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25636864 chunk: [25636864, 25640960) 25640960, 4096 00020000:00200000:3.0:1789760761.451486:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.451501:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6259, 6259] 00000008:00010000:3.0:1789760761.451515:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.451519:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.451529:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25636864, 25640960) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.451540:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25636864=0x1873000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.451585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.451588:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25640960 00000020:00010000:3.0:1789760761.451640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.451645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.451648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.451651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.451653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.451660:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.451676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.451692:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760761.451766:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25636864<>25640960 00000100:00100000:2.0:1789760761.451865:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784960:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.518162:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973578 gen 3 00000008:00010000:2.0:1789760761.518206:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760761.518213:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.518221:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.518243:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760761.518357:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493784960:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.518474:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6259, 6259] 00000080:00200000:3.0:1789760761.518487:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.518492:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.518505:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25640960 i_blocks 51207 00000020:00200000:3.0:1789760761.518508:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.518516:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.518544:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25640960, count: 4096 00000080:00200000:3.0:1789760761.518550:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25640960, bytes: 4096 00000080:00200000:3.0:1789760761.518558:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.518565:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.518567:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25640960, 25645056) 0 0 00000080:00200000:3.0:1789760761.518571:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.518584:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6260, 6260] 00020000:00200000:3.0:1789760761.518592:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25640960 chunk: [25640960, 25645056) 25645056, 4096 00020000:00200000:3.0:1789760761.518596:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.518610:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6260, 6260] 00000008:00010000:3.0:1789760761.518624:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.518628:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.518636:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25640960, 25645056) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.518648:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25640960=0x1874000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.518806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.518809:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25645056 00000020:00010000:3.0:1789760761.518826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.518830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.518833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.518837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.518839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.518846:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.518852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.518865:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760761.518938:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25640960<>25645056 00000100:00100000:2.0:1789760761.518998:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493785088:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760761.566654:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760761.566676:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.566685:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.566704:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760761.566730:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493785088:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.566868:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6260, 6260] 00000080:00200000:3.0:1789760761.566880:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.566885:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.566896:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25645056 i_blocks 51207 00000020:00200000:3.0:1789760761.566899:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.566907:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.566935:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25645056, count: 4096 00000080:00200000:3.0:1789760761.566942:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25645056, bytes: 4096 00000080:00200000:3.0:1789760761.566950:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.566958:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.566960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25645056, 25649152) 0 0 00000080:00200000:3.0:1789760761.566964:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.566977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6261, 6261] 00020000:00200000:3.0:1789760761.566984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25645056 chunk: [25645056, 25649152) 25649152, 4096 00020000:00200000:3.0:1789760761.566988:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.567001:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6261, 6261] 00000008:00010000:3.0:1789760761.567014:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.567018:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.567027:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25645056, 25649152) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.567040:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25645056=0x1875000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.567084:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.567087:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25649152 00000020:00010000:3.0:1789760761.567094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.567098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.567101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.567104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.567107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.567114:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.567120:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.567134:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760761.567223:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25645056<>25649152 00000100:00100000:2.0:1789760761.567814:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785216:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.604124:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973580 gen 3 00000008:00010000:2.0:1789760761.604165:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760761.604170:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.604177:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.604204:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760761.604230:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785216:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.604387:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6261, 6261] 00000080:00200000:3.0:1789760761.604426:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.604432:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.604443:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25649152 i_blocks 51207 00000020:00200000:3.0:1789760761.604446:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.604454:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.604542:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25649152, count: 4096 00000080:00200000:3.0:1789760761.604549:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25649152, bytes: 4096 00000080:00200000:3.0:1789760761.604557:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.604564:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.604566:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25649152, 25653248) 0 0 00000080:00200000:3.0:1789760761.604569:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.604582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6262, 6262] 00020000:00200000:3.0:1789760761.604589:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25649152 chunk: [25649152, 25653248) 25653248, 4096 00020000:00200000:3.0:1789760761.604593:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.604607:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6262, 6262] 00000008:00010000:3.0:1789760761.604621:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.604626:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.604636:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25649152, 25653248) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.604646:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25649152=0x1876000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.604695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.604698:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25653248 00000020:00010000:3.0:1789760761.604705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.604708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.604710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.604713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.604716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.604722:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.604728:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.604742:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760761.604815:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25649152<>25653248 00000100:00100000:2.0:1789760761.605152:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.605167:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785344:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.650352:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973581 gen 3 00000008:00010000:2.0:1789760761.650387:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760761.650483:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.650494:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.650516:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760761.650540:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785344:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760761.650871:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6262, 6262] 00000080:00200000:0.0:1789760761.650885:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.650890:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.650901:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25653248 i_blocks 51207 00000020:00200000:0.0:1789760761.650905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.650913:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.650940:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25653248, count: 4096 00000080:00200000:0.0:1789760761.650946:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25653248, bytes: 4096 00000080:00200000:0.0:1789760761.650954:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.650961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.650963:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25653248, 25657344) 0 0 00000080:00200000:0.0:1789760761.650967:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.650980:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6263, 6263] 00020000:00200000:0.0:1789760761.650987:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25653248 chunk: [25653248, 25657344) 25657344, 4096 00020000:00200000:0.0:1789760761.650990:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.651005:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6263, 6263] 00000008:00010000:0.0:1789760761.651019:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.651024:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760761.651033:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25653248, 25657344) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.651043:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25653248=0x1877000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.651088:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.651090:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25657344 00000020:00010000:0.0:1789760761.651096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.651099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.651102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.651105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.651108:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.651114:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.651119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.651134:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760761.651242:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 25653248<>25657344 00000100:00100000:1.0:1789760761.651464:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493785472:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760761.691882:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973582 gen 3 00000008:00010000:1.0:1789760761.691919:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760761.691926:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.691935:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.691956:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760761.691981:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493785472:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760761.692074:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6263, 6263] 00000080:00200000:0.0:1789760761.692085:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.692090:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.692101:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25657344 i_blocks 51207 00000020:00200000:0.0:1789760761.692104:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.692112:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.692136:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25657344, count: 4096 00000080:00200000:0.0:1789760761.692141:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25657344, bytes: 4096 00000080:00200000:0.0:1789760761.692148:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.692155:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.692157:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25657344, 25661440) 0 0 00000080:00200000:0.0:1789760761.692160:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.692172:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6264, 6264] 00020000:00200000:0.0:1789760761.692178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25657344 chunk: [25657344, 25661440) 25661440, 4096 00020000:00200000:0.0:1789760761.692194:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.692208:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6264, 6264] 00000008:00010000:0.0:1789760761.692220:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.692225:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760761.692233:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25657344, 25661440) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.692243:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25657344=0x1878000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.692285:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.692288:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25661440 00000020:00010000:0.0:1789760761.692294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.692298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.692300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.692303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.692305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.692311:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.692317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.692331:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760761.692436:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 25657344<>25661440 00000100:00100000:1.0:1789760761.692501:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.692512:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493785600:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760761.723312:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973583 gen 3 00000008:00010000:1.0:1789760761.723352:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760761.723360:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.723368:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.723390:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760761.723460:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493785600:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760761.723630:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6264, 6264] 00000080:00200000:0.0:1789760761.723644:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.723649:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.723661:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25661440 i_blocks 51207 00000020:00200000:0.0:1789760761.723665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.723673:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.723699:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25661440, count: 4096 00000080:00200000:0.0:1789760761.723704:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25661440, bytes: 4096 00000080:00200000:0.0:1789760761.723712:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.723719:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.723721:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25661440, 25665536) 0 0 00000080:00200000:0.0:1789760761.723724:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.723738:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6265, 6265] 00020000:00200000:0.0:1789760761.723744:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25661440 chunk: [25661440, 25665536) 25665536, 4096 00020000:00200000:0.0:1789760761.723748:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.723763:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6265, 6265] 00000008:00010000:0.0:1789760761.723775:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.723781:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760761.723789:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25661440, 25665536) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.723799:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25661440=0x1879000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.723844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.723847:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25665536 00000020:00010000:0.0:1789760761.723853:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.723857:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.723860:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.723863:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.723865:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.723871:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.723877:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.723891:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760761.724035:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 25661440<>25665536 00000100:00100000:0.0:1789760761.724079:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760761.724095:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493785728:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760761.774298:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973584 gen 3 00000008:00010000:0.0:1789760761.774334:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760761.774340:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.774348:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.774490:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6265, 6265] 00000080:00200000:3.0:1789760761.774503:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.774508:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.774521:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25665536 i_blocks 51207 00000020:00200000:3.0:1789760761.774525:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.774534:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.774561:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25665536, count: 4096 00000080:00200000:3.0:1789760761.774568:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25665536, bytes: 4096 00000080:00200000:3.0:1789760761.774576:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.774583:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.774585:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25665536, 25669632) 0 0 00000080:00200000:3.0:1789760761.774589:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.774602:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6266, 6266] 00020000:00200000:3.0:1789760761.774609:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25665536 chunk: [25665536, 25669632) 25669632, 4096 00020000:00200000:3.0:1789760761.774613:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.774626:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6266, 6266] 00000008:00010000:3.0:1789760761.774637:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.774641:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.774650:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25665536, 25669632) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.774671:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25665536=0x187a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.774715:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.774717:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25669632 00000020:00010000:3.0:1789760761.774723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.774726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.774729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.774731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.774734:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.774740:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.774745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.774910:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760761.774988:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25665536<>25669632 00000100:00100000:2.0:1789760761.775065:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785856:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760761.775693:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760761.775717:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493785728:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.830983:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973585 gen 3 00000008:00010000:2.0:1789760761.831019:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760761.831025:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.831034:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.831054:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760761.831079:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785856:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.831176:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6266, 6266] 00000080:00200000:3.0:1789760761.831197:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.831202:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.831214:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25669632 i_blocks 51207 00000020:00200000:3.0:1789760761.831217:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.831226:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.831251:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25669632, count: 4096 00000080:00200000:3.0:1789760761.831257:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25669632, bytes: 4096 00000080:00200000:3.0:1789760761.831265:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.831272:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.831274:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25669632, 25673728) 0 0 00000080:00200000:3.0:1789760761.831278:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.831290:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6267, 6267] 00020000:00200000:3.0:1789760761.831297:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25669632 chunk: [25669632, 25673728) 25673728, 4096 00020000:00200000:3.0:1789760761.831300:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.831315:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6267, 6267] 00000008:00010000:3.0:1789760761.831329:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.831333:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.831342:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25669632, 25673728) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.831354:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25669632=0x187b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.831471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.831473:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25673728 00000020:00010000:3.0:1789760761.831479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.831481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.831484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.831486:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.831488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.831492:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.831496:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.831507:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760761.831572:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25669632<>25673728 00000100:00100000:2.0:1789760761.831634:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.831645:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785984:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.882367:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973586 gen 3 00000008:00010000:2.0:1789760761.882435:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760761.882442:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.882451:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.882472:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760761.882498:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493785984:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.882646:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6267, 6267] 00000080:00200000:3.0:1789760761.882667:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.882672:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.882685:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25673728 i_blocks 51207 00000020:00200000:3.0:1789760761.882689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.882697:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.882723:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25673728, count: 4096 00000080:00200000:3.0:1789760761.882729:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25673728, bytes: 4096 00000080:00200000:3.0:1789760761.882737:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.882937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.882940:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25673728, 25677824) 0 0 00000080:00200000:3.0:1789760761.882945:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.882958:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6268, 6268] 00020000:00200000:3.0:1789760761.882964:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25673728 chunk: [25673728, 25677824) 25677824, 4096 00020000:00200000:3.0:1789760761.882968:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.882983:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6268, 6268] 00000008:00010000:3.0:1789760761.882995:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.883000:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.883009:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25673728, 25677824) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.883019:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25673728=0x187c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.883062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.883064:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25677824 00000020:00010000:3.0:1789760761.883071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.883074:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.883076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.883080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.883082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.883088:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.883094:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.883110:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760761.883196:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25673728<>25677824 00000100:00100000:2.0:1789760761.883321:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786112:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760761.935252:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973587 gen 3 00000008:00010000:2.0:1789760761.935290:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760761.935296:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760761.935304:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760761.935328:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760761.935352:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786112:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760761.935473:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6268, 6268] 00000080:00200000:3.0:1789760761.935485:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.935490:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.935502:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25677824 i_blocks 51207 00000020:00200000:3.0:1789760761.935505:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.935514:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.935541:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25677824, count: 4096 00000080:00200000:3.0:1789760761.935546:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25677824, bytes: 4096 00000080:00200000:3.0:1789760761.935554:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.935562:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.935564:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25677824, 25681920) 0 0 00000080:00200000:3.0:1789760761.935569:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.935582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6269, 6269] 00020000:00200000:3.0:1789760761.935590:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25677824 chunk: [25677824, 25681920) 25681920, 4096 00020000:00200000:3.0:1789760761.935593:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.935607:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6269, 6269] 00000008:00010000:3.0:1789760761.935622:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.935627:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760761.935635:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25677824, 25681920) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.935646:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25677824=0x187d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.935701:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.935704:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25681920 00000020:00010000:3.0:1789760761.935711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.935714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.935716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.935719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.935722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.935728:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.935733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.935755:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760761.935830:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25677824<>25681920 00000100:00100000:2.0:1789760761.936004:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.936020:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786240:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.013931:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973588 gen 3 00000008:00010000:2.0:1789760762.013968:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760762.013976:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.013984:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.014004:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760762.014024:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786240:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.015028:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6269, 6269] 00000080:00200000:3.0:1789760762.015042:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.015047:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.015058:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25681920 i_blocks 51207 00000020:00200000:3.0:1789760762.015061:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.015070:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.015094:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25681920, count: 4096 00000080:00200000:3.0:1789760762.015100:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25681920, bytes: 4096 00000080:00200000:3.0:1789760762.015109:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.015116:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.015118:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25681920, 25686016) 0 0 00000080:00200000:3.0:1789760762.015122:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.015134:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6270, 6270] 00020000:00200000:3.0:1789760762.015141:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25681920 chunk: [25681920, 25686016) 25686016, 4096 00020000:00200000:3.0:1789760762.015145:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.015159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6270, 6270] 00000008:00010000:3.0:1789760762.015174:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.015191:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.015200:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25681920, 25686016) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.015213:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25681920=0x187e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.015257:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.015259:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25686016 00000020:00010000:3.0:1789760762.015265:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.015268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.015271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.015274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.015276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.015283:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.015288:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.015301:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760762.015374:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25681920<>25686016 00000100:00100000:2.0:1789760762.016811:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786368:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760762.094666:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760762.105728:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973589 gen 3 00000008:00010000:2.0:1789760762.105772:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760762.105778:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.105786:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.105811:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760762.105841:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786368:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.106106:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6270, 6270] 00000080:00200000:3.0:1789760762.106118:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.106124:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.106135:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25686016 i_blocks 51207 00000020:00200000:3.0:1789760762.106138:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.106148:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.106174:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25686016, count: 4096 00000080:00200000:3.0:1789760762.106190:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25686016, bytes: 4096 00000080:00200000:3.0:1789760762.106202:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.106217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.106220:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25686016, 25690112) 0 0 00000080:00200000:3.0:1789760762.106224:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.106236:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6271, 6271] 00020000:00200000:3.0:1789760762.106243:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25686016 chunk: [25686016, 25690112) 25690112, 4096 00020000:00200000:3.0:1789760762.106246:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.106262:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6271, 6271] 00000008:00010000:3.0:1789760762.106285:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.106290:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.106300:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25686016, 25690112) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.106312:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25686016=0x187f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.106359:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.106362:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25690112 00000020:00010000:3.0:1789760762.106370:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.106373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.106375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.106378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.106381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.106387:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.106426:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.106442:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760762.106525:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25686016<>25690112 00000100:00100000:2.0:1789760762.106812:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.106829:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786496:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.163640:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973590 gen 3 00000008:00010000:2.0:1789760762.163678:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760762.163685:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.163693:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.163712:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760762.163737:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786496:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.163784:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6271, 6271] 00000080:00200000:3.0:1789760762.163798:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.163804:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.163832:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25690112 i_blocks 51207 00000020:00200000:3.0:1789760762.163837:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.163844:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.163872:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25690112, count: 4096 00000080:00200000:3.0:1789760762.163878:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25690112, bytes: 4096 00000080:00200000:3.0:1789760762.163886:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.163893:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.163895:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25690112, 25694208) 0 0 00000080:00200000:3.0:1789760762.163899:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.163912:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6272, 6272] 00020000:00200000:3.0:1789760762.163918:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25690112 chunk: [25690112, 25694208) 25694208, 4096 00020000:00200000:3.0:1789760762.163922:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.163939:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6272, 6272] 00000008:00010000:3.0:1789760762.163953:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.163957:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.163966:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25690112, 25694208) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.163977:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25690112=0x1880000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.164028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.164031:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25694208 00000020:00010000:3.0:1789760762.164039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.164042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.164045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.164048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.164051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.164057:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.164063:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.164078:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760762.164158:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25690112<>25694208 00000100:00100000:2.0:1789760762.164446:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786624:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.199153:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973591 gen 3 00000008:00010000:2.0:1789760762.199202:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760762.199209:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.199217:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.199336:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760762.199365:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786624:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.199511:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6272, 6272] 00000080:00200000:3.0:1789760762.199523:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.199529:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.199542:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25694208 i_blocks 51207 00000020:00200000:3.0:1789760762.199545:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.199553:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.199582:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25694208, count: 4096 00000080:00200000:3.0:1789760762.199588:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25694208, bytes: 4096 00000080:00200000:3.0:1789760762.199598:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.199605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.199607:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25694208, 25698304) 0 0 00000080:00200000:3.0:1789760762.199611:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.199627:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6273, 6273] 00020000:00200000:3.0:1789760762.199633:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25694208 chunk: [25694208, 25698304) 25698304, 4096 00020000:00200000:3.0:1789760762.199637:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.199653:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6273, 6273] 00000008:00010000:3.0:1789760762.199675:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.199680:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.199690:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25694208, 25698304) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.199702:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25694208=0x1881000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.199745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.199748:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25698304 00000020:00010000:3.0:1789760762.199754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.199757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.199760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.199763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.199765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.199771:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.199775:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.199788:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760762.199922:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25694208<>25698304 00000100:00100000:2.0:1789760762.200002:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.200016:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786752:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.248105:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973592 gen 3 00000008:00010000:2.0:1789760762.248141:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760762.248147:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.248155:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.248173:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760762.248207:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786752:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.248357:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6273, 6273] 00000080:00200000:3.0:1789760762.248370:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.248380:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.248430:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25698304 i_blocks 51207 00000020:00200000:3.0:1789760762.248434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.248442:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.248476:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25698304, count: 4096 00000080:00200000:3.0:1789760762.248483:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25698304, bytes: 4096 00000080:00200000:3.0:1789760762.248491:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.248498:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.248500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25698304, 25702400) 0 0 00000080:00200000:3.0:1789760762.248507:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.248520:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6274, 6274] 00020000:00200000:3.0:1789760762.248526:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25698304 chunk: [25698304, 25702400) 25702400, 4096 00020000:00200000:3.0:1789760762.248529:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.248544:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6274, 6274] 00000008:00010000:3.0:1789760762.248556:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.248560:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.248568:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25698304, 25702400) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.248578:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25698304=0x1882000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.248619:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.248622:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25702400 00000020:00010000:3.0:1789760762.248628:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.248631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.248634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.248636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.248639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.248645:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.248651:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.248729:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760762.248806:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25698304<>25702400 00000100:00100000:2.0:1789760762.248915:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786880:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.286876:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973593 gen 3 00000008:00010000:2.0:1789760762.286913:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760762.286920:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.286928:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.286946:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760762.286972:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493786880:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.287385:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6274, 6274] 00000080:00200000:3.0:1789760762.287418:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.287425:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.287436:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25702400 i_blocks 51207 00000020:00200000:3.0:1789760762.287440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.287449:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.287477:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25702400, count: 4096 00000080:00200000:3.0:1789760762.287483:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25702400, bytes: 4096 00000080:00200000:3.0:1789760762.287492:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.287499:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.287501:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25702400, 25706496) 0 0 00000080:00200000:3.0:1789760762.287505:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.287517:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6275, 6275] 00020000:00200000:3.0:1789760762.287524:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25702400 chunk: [25702400, 25706496) 25706496, 4096 00020000:00200000:3.0:1789760762.287528:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.287542:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6275, 6275] 00000008:00010000:3.0:1789760762.287555:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.287559:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.287568:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25702400, 25706496) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.287580:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25702400=0x1883000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.287623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.287625:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25706496 00000020:00010000:3.0:1789760762.287632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.287635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.287637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.287640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.287643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.287649:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.287710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.287728:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760762.287808:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25702400<>25706496 00000100:00100000:2.0:1789760762.287909:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.287925:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493787008:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.329334:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973594 gen 3 00000008:00010000:2.0:1789760762.329372:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760762.329379:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.329388:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.329443:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760762.329470:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493787008:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.329565:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6275, 6275] 00000080:00200000:3.0:1789760762.329577:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.329583:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.329594:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25706496 i_blocks 51207 00000020:00200000:3.0:1789760762.329597:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.329607:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.329634:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25706496, count: 4096 00000080:00200000:3.0:1789760762.329641:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25706496, bytes: 4096 00000080:00200000:3.0:1789760762.329649:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.329723:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.329726:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25706496, 25710592) 0 0 00000080:00200000:3.0:1789760762.329730:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.329744:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6276, 6276] 00020000:00200000:3.0:1789760762.329751:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25706496 chunk: [25706496, 25710592) 25710592, 4096 00020000:00200000:3.0:1789760762.329755:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.329772:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6276, 6276] 00000008:00010000:3.0:1789760762.329786:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.329791:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.329799:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25706496, 25710592) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.329810:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25706496=0x1884000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.329866:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.329869:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25710592 00000020:00010000:3.0:1789760762.329876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.329879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.329882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.329885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.329887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.329893:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.329899:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.329912:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760762.329994:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25706496<>25710592 00000100:00100000:2.0:1789760762.330110:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493787136:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.385454:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973595 gen 3 00000008:00010000:2.0:1789760762.385493:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760762.385499:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.385507:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.385528:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760762.385553:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493787136:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.392270:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6276, 6276] 00000080:00200000:3.0:1789760762.392291:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.392296:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.392308:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25710592 i_blocks 51207 00000020:00200000:3.0:1789760762.392313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.392324:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.392351:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25710592, count: 4096 00000080:00200000:3.0:1789760762.392358:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25710592, bytes: 4096 00000080:00200000:3.0:1789760762.392370:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.392377:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.392379:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25710592, 25714688) 0 0 00000080:00200000:3.0:1789760762.392383:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.392426:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6277, 6277] 00020000:00200000:3.0:1789760762.392433:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25710592 chunk: [25710592, 25714688) 25714688, 4096 00020000:00200000:3.0:1789760762.392437:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.392450:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6277, 6277] 00000008:00010000:3.0:1789760762.392464:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.392468:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.392478:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25710592, 25714688) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.392489:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25710592=0x1885000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.392543:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.392545:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25714688 00000020:00010000:3.0:1789760762.392552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.392555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.392559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.392562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.392564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.392571:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.392575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.392594:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760762.392678:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25710592<>25714688 00000100:00100000:2.0:1789760762.398478:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787264:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760762.413229:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493787392:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.413240:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787520:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.413355:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493787648:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.426615:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787520:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.426664:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493787392:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.426677:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493787648:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760762.460990:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973596 gen 3 00000008:00010000:2.0:1789760762.461023:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760762.461030:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.461037:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.461057:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760762.461082:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787264:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.461243:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6277, 6277] 00000080:00200000:3.0:1789760762.461255:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.461260:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.461271:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25714688 i_blocks 51207 00000020:00200000:3.0:1789760762.461274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.461282:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.461306:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25714688, count: 4096 00000080:00200000:3.0:1789760762.461311:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25714688, bytes: 4096 00000080:00200000:3.0:1789760762.461318:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.461324:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.461326:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25714688, 25718784) 0 0 00000080:00200000:3.0:1789760762.461330:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.461342:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6278, 6278] 00020000:00200000:3.0:1789760762.461349:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25714688 chunk: [25714688, 25718784) 25718784, 4096 00020000:00200000:3.0:1789760762.461352:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.461557:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6278, 6278] 00000008:00010000:3.0:1789760762.461572:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.461576:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.461586:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25714688, 25718784) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.461596:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25714688=0x1886000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.461636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.461639:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25718784 00000020:00010000:3.0:1789760762.461645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.461648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.461652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.461663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.461666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.461672:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.461679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.461693:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760762.461766:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25714688<>25718784 00000100:00100000:2.0:1789760762.462127:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787776:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.510302:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973597 gen 3 00000008:00010000:2.0:1789760762.510340:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760762.510347:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.510356:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.510382:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760762.510618:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787776:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.510771:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6278, 6278] 00000080:00200000:3.0:1789760762.510784:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.510789:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.510801:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25718784 i_blocks 51207 00000020:00200000:3.0:1789760762.510805:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.510813:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.510844:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25718784, count: 4096 00000080:00200000:3.0:1789760762.510849:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25718784, bytes: 4096 00000080:00200000:3.0:1789760762.510859:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.510866:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.510868:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25718784, 25722880) 0 0 00000080:00200000:3.0:1789760762.510871:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.510884:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6279, 6279] 00020000:00200000:3.0:1789760762.510891:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25718784 chunk: [25718784, 25722880) 25722880, 4096 00020000:00200000:3.0:1789760762.510895:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.510909:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6279, 6279] 00000008:00010000:3.0:1789760762.510923:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.510928:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.510937:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25718784, 25722880) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.510946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25718784=0x1887000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.510989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.510992:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25722880 00000020:00010000:3.0:1789760762.510999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.511002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.511004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.511007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.511010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.511016:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.511022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.511036:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760762.511115:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25718784<>25722880 00000100:00100000:2.0:1789760762.511743:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760762.511758:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787904:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.568341:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973598 gen 3 00000008:00010000:2.0:1789760762.568375:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760762.568381:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.568391:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.568451:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760762.568474:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493787904:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.568637:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6279, 6279] 00000080:00200000:3.0:1789760762.568650:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.568655:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.568667:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25722880 i_blocks 51207 00000020:00200000:3.0:1789760762.568671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.568678:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.568721:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25722880, count: 4096 00000080:00200000:3.0:1789760762.568727:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25722880, bytes: 4096 00000080:00200000:3.0:1789760762.568738:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.568745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.568747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25722880, 25726976) 0 0 00000080:00200000:3.0:1789760762.568751:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.568763:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6280, 6280] 00020000:00200000:3.0:1789760762.568769:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25722880 chunk: [25722880, 25726976) 25726976, 4096 00020000:00200000:3.0:1789760762.568773:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.568789:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6280, 6280] 00000008:00010000:3.0:1789760762.568804:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.568808:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.568818:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25722880, 25726976) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.568830:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25722880=0x1888000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.568872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.568875:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25726976 00000020:00010000:3.0:1789760762.568882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.568885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.568888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.568892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.568895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.568900:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.568906:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.568919:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760762.569920:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25722880<>25726976 00000100:00100000:2.0:1789760762.570081:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.570099:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493788032:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.676377:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973599 gen 3 00000008:00010000:2.0:1789760762.676452:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760762.676459:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.676468:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.676494:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760762.676521:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493788032:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.676711:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6280, 6280] 00000080:00200000:3.0:1789760762.676724:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.676730:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.676742:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25726976 i_blocks 51207 00000020:00200000:3.0:1789760762.676745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.676754:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.676783:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25726976, count: 4096 00000080:00200000:3.0:1789760762.676790:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25726976, bytes: 4096 00000080:00200000:3.0:1789760762.676800:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.676807:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.676810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25726976, 25731072) 0 0 00000080:00200000:3.0:1789760762.676814:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.676826:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6281, 6281] 00020000:00200000:3.0:1789760762.676833:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25726976 chunk: [25726976, 25731072) 25731072, 4096 00020000:00200000:3.0:1789760762.676837:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.676853:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6281, 6281] 00000008:00010000:3.0:1789760762.676867:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.676872:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.676881:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25726976, 25731072) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.676892:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25726976=0x1889000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.676935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.676938:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25731072 00000020:00010000:3.0:1789760762.676945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.676948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.676951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.676954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.676957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.676963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.676969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.676984:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760762.677059:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25726976<>25731072 00000100:00100000:2.0:1789760762.677216:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760762.677231:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788160:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.758510:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973600 gen 3 00000008:00010000:2.0:1789760762.758548:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760762.758554:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.758562:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.758582:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760762.758607:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788160:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.758775:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6281, 6281] 00000080:00200000:3.0:1789760762.758788:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.758792:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.758803:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25731072 i_blocks 51207 00000020:00200000:3.0:1789760762.758806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.758825:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.758853:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25731072, count: 4096 00000080:00200000:3.0:1789760762.758859:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25731072, bytes: 4096 00000080:00200000:3.0:1789760762.758868:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.758875:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.758877:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25731072, 25735168) 0 0 00000080:00200000:3.0:1789760762.758881:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.758893:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6282, 6282] 00020000:00200000:3.0:1789760762.758899:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25731072 chunk: [25731072, 25735168) 25735168, 4096 00020000:00200000:3.0:1789760762.758903:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.758917:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6282, 6282] 00000008:00010000:3.0:1789760762.758931:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.758936:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.758944:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25731072, 25735168) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.758955:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25731072=0x188a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.759142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.759145:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25735168 00000020:00010000:3.0:1789760762.759152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.759155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.759157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.759160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.759163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.759170:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.759186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.759202:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760762.759360:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25731072<>25735168 00000100:00100000:2.0:1789760762.759540:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.759556:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493788288:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760762.822597:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760762.822607:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.822616:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.822635:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760762.822668:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493788288:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.824120:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6282, 6282] 00000080:00200000:3.0:1789760762.824136:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.824141:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.824153:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25735168 i_blocks 51207 00000020:00200000:3.0:1789760762.824157:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.824167:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.824206:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25735168, count: 4096 00000080:00200000:3.0:1789760762.824212:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25735168, bytes: 4096 00000080:00200000:3.0:1789760762.824223:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.824230:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.824232:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25735168, 25739264) 0 0 00000080:00200000:3.0:1789760762.824236:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.824248:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6283, 6283] 00020000:00200000:3.0:1789760762.824255:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25735168 chunk: [25735168, 25739264) 25739264, 4096 00020000:00200000:3.0:1789760762.824259:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.824276:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6283, 6283] 00000008:00010000:3.0:1789760762.824289:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.824294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.824303:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25735168, 25739264) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.824314:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25735168=0x188b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.824364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.824366:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25739264 00000020:00010000:3.0:1789760762.824373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.824376:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.824379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.824382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.824384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.824434:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.824441:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.824459:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760762.824542:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25735168<>25739264 00000100:00100000:2.0:1789760762.824630:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760762.824720:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788416:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760762.868447:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973602 gen 3 00000008:00010000:2.0:1789760762.868497:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760762.868503:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.868512:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.868531:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760762.868556:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788416:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760762.868588:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6283, 6283] 00000080:00200000:3.0:1789760762.868600:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.868606:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.868618:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25739264 i_blocks 51207 00000020:00200000:3.0:1789760762.868621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.868753:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.868784:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25739264, count: 4096 00000080:00200000:3.0:1789760762.868791:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25739264, bytes: 4096 00000080:00200000:3.0:1789760762.868800:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.868808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.868810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25739264, 25743360) 0 0 00000080:00200000:3.0:1789760762.868815:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.868828:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6284, 6284] 00020000:00200000:3.0:1789760762.868836:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25739264 chunk: [25739264, 25743360) 25743360, 4096 00020000:00200000:3.0:1789760762.868840:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.868856:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6284, 6284] 00000008:00010000:3.0:1789760762.868871:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.868876:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760762.868886:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25739264, 25743360) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.868896:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25739264=0x188c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.868942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.868945:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25743360 00000020:00010000:3.0:1789760762.868951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.868954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.868957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.868961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.868963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.868969:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.868975:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.868990:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760762.869071:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25739264<>25743360 00000100:00100000:2.0:1789760762.869202:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788544:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760762.923923:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760762.923937:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760762.923948:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760762.923969:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760762.923996:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493788544:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760762.924091:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6284, 6284] 00000080:00200000:0.0:1789760762.924104:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.924110:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.924124:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25743360 i_blocks 51207 00000020:00200000:0.0:1789760762.924128:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.924136:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.924168:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25743360, count: 4096 00000080:00200000:0.0:1789760762.924183:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25743360, bytes: 4096 00000080:00200000:0.0:1789760762.924193:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.924201:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.924204:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25743360, 25747456) 0 0 00000080:00200000:0.0:1789760762.924208:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.924233:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6285, 6285] 00020000:00200000:0.0:1789760762.924241:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25743360 chunk: [25743360, 25747456) 25747456, 4096 00020000:00200000:0.0:1789760762.924245:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.924262:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6285, 6285] 00000008:00010000:0.0:1789760762.924273:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.924278:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760762.924287:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25743360, 25747456) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.924298:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25743360=0x188d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.924343:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.924345:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25747456 00000020:00010000:0.0:1789760762.924351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.924354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.924357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.924360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.924362:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.924368:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.924373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.924387:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760762.924501:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 25743360<>25747456 00000100:00100000:1.0:1789760762.924587:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760762.924601:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760762.969244:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973604 gen 3 00000008:00010000:1.0:1789760762.969292:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760762.969299:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.969306:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.969327:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760762.969353:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788672:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760762.969613:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6285, 6285] 00000080:00200000:0.0:1789760762.969635:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.969641:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.969653:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25747456 i_blocks 51207 00000020:00200000:0.0:1789760762.969657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.969667:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.969695:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25747456, count: 4096 00000080:00200000:0.0:1789760762.969701:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25747456, bytes: 4096 00000080:00200000:0.0:1789760762.969711:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.969718:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.969720:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25747456, 25751552) 0 0 00000080:00200000:0.0:1789760762.969724:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.969738:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6286, 6286] 00020000:00200000:0.0:1789760762.969744:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25747456 chunk: [25747456, 25751552) 25751552, 4096 00020000:00200000:0.0:1789760762.969748:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.969763:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6286, 6286] 00000008:00010000:0.0:1789760762.969776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.969780:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760762.969788:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25747456, 25751552) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.969798:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25747456=0x188e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.969839:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.969842:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25751552 00000020:00010000:0.0:1789760762.969848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.969851:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.969854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.969857:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.969860:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.969865:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.969870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.969885:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760762.969958:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 25747456<>25751552 00000100:00100000:1.0:1789760762.970119:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788800:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760763.050706:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760763.050716:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.050725:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.050745:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760763.050773:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788800:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.050871:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6286, 6286] 00000080:00200000:0.0:1789760763.050883:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.050888:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.050899:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25751552 i_blocks 51207 00000020:00200000:0.0:1789760763.050902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.050911:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.050938:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25751552, count: 4096 00000080:00200000:0.0:1789760763.050944:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25751552, bytes: 4096 00000080:00200000:0.0:1789760763.050952:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.050959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.050961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25751552, 25755648) 0 0 00000080:00200000:0.0:1789760763.050965:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.050977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6287, 6287] 00020000:00200000:0.0:1789760763.050984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25751552 chunk: [25751552, 25755648) 25755648, 4096 00020000:00200000:0.0:1789760763.050987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.051001:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6287, 6287] 00000008:00010000:0.0:1789760763.051013:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.051018:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.051026:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25751552, 25755648) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.051038:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25751552=0x188f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.051079:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.051082:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25755648 00000020:00010000:0.0:1789760763.051089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.051092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.051094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.051098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.051100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.051106:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.051112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.051125:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760763.051219:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 25751552<>25755648 00000100:00100000:1.0:1789760763.051300:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.051314:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.111076:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973606 gen 3 00000008:00010000:1.0:1789760763.111123:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760763.111129:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.111136:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.111158:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760763.111194:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493788928:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.111732:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6287, 6287] 00000080:00200000:0.0:1789760763.111746:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.111750:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.111772:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25755648 i_blocks 51207 00000020:00200000:0.0:1789760763.111777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.111786:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.111814:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25755648, count: 4096 00000080:00200000:0.0:1789760763.111819:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25755648, bytes: 4096 00000080:00200000:0.0:1789760763.111828:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.111834:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.111836:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25755648, 25759744) 0 0 00000080:00200000:0.0:1789760763.111839:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.111851:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6288, 6288] 00020000:00200000:0.0:1789760763.111857:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25755648 chunk: [25755648, 25759744) 25759744, 4096 00020000:00200000:0.0:1789760763.111860:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.111875:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6288, 6288] 00000008:00010000:0.0:1789760763.111888:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.111893:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.111901:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25755648, 25759744) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.111912:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25755648=0x1890000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.111958:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.111961:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25759744 00000020:00010000:0.0:1789760763.111968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.111971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.111974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.111977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.111979:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.111986:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.111991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.112006:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760763.112082:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 25755648<>25759744 00000100:00100000:1.0:1789760763.112246:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789056:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760763.116769:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760763.195267:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973607 gen 3 00000008:00010000:1.0:1789760763.195304:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760763.195311:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.195319:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.195339:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760763.195365:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789056:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.195722:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6288, 6288] 00000080:00200000:0.0:1789760763.195735:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.195739:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.195751:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25759744 i_blocks 51207 00000020:00200000:0.0:1789760763.195756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.195764:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.195791:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25759744, count: 4096 00000080:00200000:0.0:1789760763.195797:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25759744, bytes: 4096 00000080:00200000:0.0:1789760763.195805:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.195812:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.195814:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25759744, 25763840) 0 0 00000080:00200000:0.0:1789760763.195818:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.195831:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6289, 6289] 00020000:00200000:0.0:1789760763.195837:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25759744 chunk: [25759744, 25763840) 25763840, 4096 00020000:00200000:0.0:1789760763.195840:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.195854:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6289, 6289] 00000008:00010000:0.0:1789760763.195878:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.195883:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.195902:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25759744, 25763840) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.195913:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25759744=0x1891000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.201815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.201824:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25763840 00000020:00010000:0.0:1789760763.201837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.201841:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.201845:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.201849:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.201852:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.201863:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.201871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.201972:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760763.202064:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 25759744<>25763840 00000100:00100000:0.0:1789760763.202106:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760763.202124:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789184:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760763.265069:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973608 gen 3 00000008:00010000:0.0:1789760763.265105:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760763.265112:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.265120:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.265149:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760763.265184:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789184:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760763.265368:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6289, 6289] 00000080:00200000:2.0:1789760763.265381:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.265386:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.265426:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25763840 i_blocks 51207 00000020:00200000:2.0:1789760763.265431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.265440:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.265466:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25763840, count: 4096 00000080:00200000:2.0:1789760763.265471:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25763840, bytes: 4096 00000080:00200000:2.0:1789760763.265481:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.265488:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.265490:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25763840, 25767936) 0 0 00000080:00200000:2.0:1789760763.265493:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.265506:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6290, 6290] 00020000:00200000:2.0:1789760763.265512:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25763840 chunk: [25763840, 25767936) 25767936, 4096 00020000:00200000:2.0:1789760763.265516:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.265530:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6290, 6290] 00000008:00010000:2.0:1789760763.265543:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.265548:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760763.265556:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25763840, 25767936) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.265567:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25763840=0x1892000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.265609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.265612:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25767936 00000020:00010000:2.0:1789760763.265618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.265622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.265727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.265730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.265733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.265741:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.265748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.265762:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760763.265848:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 25763840<>25767936 00000100:00100000:2.0:1789760763.265887:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760763.265902:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493789312:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760763.344889:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973609 gen 3 00000008:00010000:2.0:1789760763.344924:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760763.344930:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760763.344940:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760763.344977:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760763.345001:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493789312:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.345221:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6290, 6290] 00000080:00200000:0.0:1789760763.345234:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.345239:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.345251:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25767936 i_blocks 51207 00000020:00200000:0.0:1789760763.345254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.345262:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.345301:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25767936, count: 4096 00000080:00200000:0.0:1789760763.345308:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25767936, bytes: 4096 00000080:00200000:0.0:1789760763.345318:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.345325:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.345328:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25767936, 25772032) 0 0 00000080:00200000:0.0:1789760763.345332:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.345344:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6291, 6291] 00020000:00200000:0.0:1789760763.345352:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25767936 chunk: [25767936, 25772032) 25772032, 4096 00020000:00200000:0.0:1789760763.345356:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.345372:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6291, 6291] 00000008:00010000:0.0:1789760763.345387:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.345419:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.345431:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25767936, 25772032) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.345441:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25767936=0x1893000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.345488:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.345491:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25772032 00000020:00010000:0.0:1789760763.345497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.345500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.345503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.345506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.345509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.345515:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.345520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.345535:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760763.345613:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 25767936<>25772032 00000100:00100000:1.0:1789760763.345981:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.345998:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789440:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760763.418331:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760763.418344:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.418353:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.418379:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760763.418449:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789440:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.418589:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6291, 6291] 00000080:00200000:0.0:1789760763.418600:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.418606:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.418617:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25772032 i_blocks 51207 00000020:00200000:0.0:1789760763.418621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.418629:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.418654:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25772032, count: 4096 00000080:00200000:0.0:1789760763.418660:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25772032, bytes: 4096 00000080:00200000:0.0:1789760763.418673:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.418681:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.418683:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25772032, 25776128) 0 0 00000080:00200000:0.0:1789760763.418688:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.418700:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6292, 6292] 00020000:00200000:0.0:1789760763.418706:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25772032 chunk: [25772032, 25776128) 25776128, 4096 00020000:00200000:0.0:1789760763.418710:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.418726:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6292, 6292] 00000008:00010000:0.0:1789760763.418739:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.418744:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.418753:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25772032, 25776128) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.418764:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25772032=0x1894000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.418811:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.418814:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25776128 00000020:00010000:0.0:1789760763.418820:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.418823:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.418826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.418830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.418833:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.418839:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.418844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.418859:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760763.418938:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 25772032<>25776128 00000100:00100000:1.0:1789760763.419608:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.419627:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789568:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.466257:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973611 gen 3 00000008:00010000:1.0:1789760763.466308:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760763.466315:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.466322:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.466343:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760763.466381:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789568:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.466750:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6292, 6292] 00000080:00200000:0.0:1789760763.466764:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.466769:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.466781:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25776128 i_blocks 51207 00000020:00200000:0.0:1789760763.466784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.466793:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.466818:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25776128, count: 4096 00000080:00200000:0.0:1789760763.466824:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25776128, bytes: 4096 00000080:00200000:0.0:1789760763.466835:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.466842:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.466844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25776128, 25780224) 0 0 00000080:00200000:0.0:1789760763.466848:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.466860:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6293, 6293] 00020000:00200000:0.0:1789760763.466867:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25776128 chunk: [25776128, 25780224) 25780224, 4096 00020000:00200000:0.0:1789760763.466871:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.466885:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6293, 6293] 00000008:00010000:0.0:1789760763.466899:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.466904:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.466913:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25776128, 25780224) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.466924:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25776128=0x1895000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.467040:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.467043:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25780224 00000020:00010000:0.0:1789760763.467051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.467054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.467057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.467061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.467064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.467070:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.467077:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.467457:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760763.467541:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 25776128<>25780224 00000100:00100000:1.0:1789760763.467715:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.467732:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789696:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.528532:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973612 gen 3 00000008:00010000:1.0:1789760763.528569:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760763.528577:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.528586:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.528610:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760763.528641:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789696:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.528759:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6293, 6293] 00000080:00200000:0.0:1789760763.528772:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.528777:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.528789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25780224 i_blocks 51207 00000020:00200000:0.0:1789760763.528793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.528802:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.528830:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25780224, count: 4096 00000080:00200000:0.0:1789760763.528835:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25780224, bytes: 4096 00000080:00200000:0.0:1789760763.528846:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.528853:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.528855:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25780224, 25784320) 0 0 00000080:00200000:0.0:1789760763.528859:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.528872:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6294, 6294] 00020000:00200000:0.0:1789760763.528879:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25780224 chunk: [25780224, 25784320) 25784320, 4096 00020000:00200000:0.0:1789760763.528883:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.528899:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6294, 6294] 00000008:00010000:0.0:1789760763.528913:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.528918:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.528927:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25780224, 25784320) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.528937:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25780224=0x1896000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.528993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.528996:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25784320 00000020:00010000:0.0:1789760763.529002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.529006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.529008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.529012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.529015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.529021:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.529026:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.529042:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760763.529118:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 25780224<>25784320 00000100:00100000:1.0:1789760763.529851:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.529870:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789824:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760763.574620:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760763.574636:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.574648:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.574744:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760763.574778:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493789824:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.574927:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6294, 6294] 00000080:00200000:0.0:1789760763.574944:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.574951:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.574967:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25784320 i_blocks 51207 00000020:00200000:0.0:1789760763.574971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.574981:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.575014:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25784320, count: 4096 00000080:00200000:0.0:1789760763.575023:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25784320, bytes: 4096 00000080:00200000:0.0:1789760763.575035:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.575044:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.575046:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25784320, 25788416) 0 0 00000080:00200000:0.0:1789760763.575050:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.575067:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6295, 6295] 00020000:00200000:0.0:1789760763.575077:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25784320 chunk: [25784320, 25788416) 25788416, 4096 00020000:00200000:0.0:1789760763.575081:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.575100:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6295, 6295] 00000008:00010000:0.0:1789760763.575119:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.575124:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.575136:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25784320, 25788416) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.575150:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25784320=0x1897000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.575220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.576589:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25788416 00000020:00010000:0.0:1789760763.576610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.576614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.576618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.576623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.576626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.576635:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.576646:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.576687:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760763.576807:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 25784320<>25788416 00000100:00100000:1.0:1789760763.577100:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.577126:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789952:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.603479:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973614 gen 3 00000008:00010000:1.0:1789760763.603547:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760763.603555:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.603565:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.603588:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760763.603618:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493789952:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.603843:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6295, 6295] 00000080:00200000:0.0:1789760763.603860:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.603866:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.603881:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25788416 i_blocks 51207 00000020:00200000:0.0:1789760763.603885:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.603895:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.603926:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25788416, count: 4096 00000080:00200000:0.0:1789760763.603932:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25788416, bytes: 4096 00000080:00200000:0.0:1789760763.603941:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.603950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.603952:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25788416, 25792512) 0 0 00000080:00200000:0.0:1789760763.603957:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.603973:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6296, 6296] 00020000:00200000:0.0:1789760763.603982:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25788416 chunk: [25788416, 25792512) 25792512, 4096 00020000:00200000:0.0:1789760763.603986:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.604003:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6296, 6296] 00000008:00010000:0.0:1789760763.604020:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.604026:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.604038:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25788416, 25792512) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.604051:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25788416=0x1898000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.604105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.604108:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25792512 00000020:00010000:0.0:1789760763.604115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.604119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.604121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.604125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.604129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.604136:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.604142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.604157:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760763.604264:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 25788416<>25792512 00000100:00100000:1.0:1789760763.604908:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.604932:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790080:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760763.657898:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760763.657916:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.657927:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.657954:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760763.657986:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790080:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.658152:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6296, 6296] 00000080:00200000:0.0:1789760763.658179:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.658186:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.658200:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25792512 i_blocks 51207 00000020:00200000:0.0:1789760763.658204:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.658214:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.658258:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25792512, count: 4096 00000080:00200000:0.0:1789760763.658266:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25792512, bytes: 4096 00000080:00200000:0.0:1789760763.658276:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.658285:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.658288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25792512, 25796608) 0 0 00000080:00200000:0.0:1789760763.658293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.658311:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6297, 6297] 00020000:00200000:0.0:1789760763.658320:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25792512 chunk: [25792512, 25796608) 25796608, 4096 00020000:00200000:0.0:1789760763.658325:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.658344:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6297, 6297] 00000008:00010000:0.0:1789760763.658362:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.658368:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.658379:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25792512, 25796608) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.658428:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25792512=0x1899000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.658482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.658485:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25796608 00000020:00010000:0.0:1789760763.658493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.658497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.658499:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.658504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.658507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.658513:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.658520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.658535:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760763.658660:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 25792512<>25796608 00000100:00100000:1.0:1789760763.659113:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.659134:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790208:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.705883:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973616 gen 3 00000008:00010000:1.0:1789760763.705952:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760763.705959:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.705969:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.705991:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760763.706020:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790208:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.706301:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6297, 6297] 00000080:00200000:0.0:1789760763.706318:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.706324:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.706340:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25796608 i_blocks 51207 00000020:00200000:0.0:1789760763.706343:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.706353:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.706388:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25796608, count: 4096 00000080:00200000:0.0:1789760763.706423:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25796608, bytes: 4096 00000080:00200000:0.0:1789760763.706434:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.706443:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.706446:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25796608, 25800704) 0 0 00000080:00200000:0.0:1789760763.706450:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.706465:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6298, 6298] 00020000:00200000:0.0:1789760763.706473:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25796608 chunk: [25796608, 25800704) 25800704, 4096 00020000:00200000:0.0:1789760763.706477:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.706496:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6298, 6298] 00000008:00010000:0.0:1789760763.706515:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.706521:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.706534:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25796608, 25800704) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.706549:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25796608=0x189a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.706604:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.706606:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25800704 00000020:00010000:0.0:1789760763.706614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.706617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.706620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.706624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.706626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.706633:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.706640:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.706667:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760763.706768:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 25796608<>25800704 00000100:00100000:1.0:1789760763.706962:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.706983:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790336:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.744600:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973617 gen 3 00000008:00010000:1.0:1789760763.744652:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760763.744661:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.744673:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.744749:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760763.744779:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790336:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.745044:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6298, 6298] 00000080:00200000:0.0:1789760763.745061:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.745162:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.745193:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25800704 i_blocks 51207 00000020:00200000:0.0:1789760763.745197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.745209:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.745245:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25800704, count: 4096 00000080:00200000:0.0:1789760763.745252:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25800704, bytes: 4096 00000080:00200000:0.0:1789760763.745262:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.745271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.745274:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25800704, 25804800) 0 0 00000080:00200000:0.0:1789760763.745279:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.745297:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6299, 6299] 00020000:00200000:0.0:1789760763.745306:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25800704 chunk: [25800704, 25804800) 25804800, 4096 00020000:00200000:0.0:1789760763.745310:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.745332:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6299, 6299] 00000008:00010000:0.0:1789760763.745349:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.745355:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.745366:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25800704, 25804800) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.745380:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25800704=0x189b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.745468:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.745472:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25804800 00000020:00010000:0.0:1789760763.745481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.745484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.745487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.745492:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.745495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.745502:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.745509:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.745526:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760763.745629:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 25800704<>25804800 00000100:00100000:1.0:1789760763.745695:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790464:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.790635:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973618 gen 3 00000008:00010000:1.0:1789760763.790703:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760763.790712:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.790722:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.790747:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760763.790778:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790464:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.790958:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6299, 6299] 00000080:00200000:0.0:1789760763.790974:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.790989:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.791004:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25804800 i_blocks 51207 00000020:00200000:0.0:1789760763.791008:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.791018:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.791050:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25804800, count: 4096 00000080:00200000:0.0:1789760763.791059:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25804800, bytes: 4096 00000080:00200000:0.0:1789760763.791069:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.791079:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.791081:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25804800, 25808896) 0 0 00000080:00200000:0.0:1789760763.791086:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.791103:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6300, 6300] 00020000:00200000:0.0:1789760763.791111:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25804800 chunk: [25804800, 25808896) 25808896, 4096 00020000:00200000:0.0:1789760763.791116:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.791138:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6300, 6300] 00000008:00010000:0.0:1789760763.791158:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.791274:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.791287:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25804800, 25808896) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.791301:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25804800=0x189c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.791347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.791350:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25808896 00000020:00010000:0.0:1789760763.791357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.791360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.791363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.791367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.791369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.791377:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.791383:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.791471:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:1.0:1789760763.791649:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 25804800<>25808896 00000100:00100000:1.0:1789760763.791687:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.791705:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790592:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.850957:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973619 gen 3 00000008:00010000:1.0:1789760763.851159:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760763.851177:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.851190:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.851214:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760763.851246:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790592:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.851383:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6300, 6300] 00000080:00200000:0.0:1789760763.851420:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.851427:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.851445:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25808896 i_blocks 51207 00000020:00200000:0.0:1789760763.851449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.851459:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.851495:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25808896, count: 4096 00000080:00200000:0.0:1789760763.851502:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25808896, bytes: 4096 00000080:00200000:0.0:1789760763.851516:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.851526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.851529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25808896, 25812992) 0 0 00000080:00200000:0.0:1789760763.851534:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.851551:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6301, 6301] 00020000:00200000:0.0:1789760763.851560:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25808896 chunk: [25808896, 25812992) 25812992, 4096 00020000:00200000:0.0:1789760763.851565:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.851587:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6301, 6301] 00000008:00010000:0.0:1789760763.851605:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.851611:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.851623:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25808896, 25812992) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.851638:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25808896=0x189d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.851772:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.851775:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25812992 00000020:00010000:0.0:1789760763.851784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.851787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.851790:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.851794:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.851797:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.851807:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.851814:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.851830:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760763.851937:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 25808896<>25812992 00000100:00100000:1.0:1789760763.852160:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.895654:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973620 gen 3 00000008:00010000:1.0:1789760763.895721:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760763.895730:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.895739:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.895764:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760763.895795:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493790720:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.896146:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6301, 6301] 00000080:00200000:0.0:1789760763.896171:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.896178:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.896195:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25812992 i_blocks 51207 00000020:00200000:0.0:1789760763.896198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.896208:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.896244:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25812992, count: 4096 00000080:00200000:0.0:1789760763.896251:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25812992, bytes: 4096 00000080:00200000:0.0:1789760763.896260:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.896269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.896271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25812992, 25817088) 0 0 00000080:00200000:0.0:1789760763.896276:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.896292:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6302, 6302] 00020000:00200000:0.0:1789760763.896301:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25812992 chunk: [25812992, 25817088) 25817088, 4096 00020000:00200000:0.0:1789760763.896305:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.896323:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6302, 6302] 00000008:00010000:0.0:1789760763.896342:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.896349:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.896361:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25812992, 25817088) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.896375:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25812992=0x189e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.896460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.896463:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25817088 00000020:00010000:0.0:1789760763.896472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.896476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.896479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.896483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.896485:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.896493:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.896498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.896511:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760763.896597:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 25812992<>25817088 00000100:00100000:1.0:1789760763.896802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790848:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.935457:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973621 gen 3 00000008:00010000:1.0:1789760763.935511:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760763.935520:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.935531:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.935544:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760763.935571:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790848:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.936216:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6302, 6302] 00000080:00200000:0.0:1789760763.936235:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.936241:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.936255:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25817088 i_blocks 51207 00000020:00200000:0.0:1789760763.936259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.936271:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.936305:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25817088, count: 4096 00000080:00200000:0.0:1789760763.936313:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25817088, bytes: 4096 00000080:00200000:0.0:1789760763.936327:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.936335:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.936337:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25817088, 25821184) 0 0 00000080:00200000:0.0:1789760763.936342:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.936359:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6303, 6303] 00020000:00200000:0.0:1789760763.936368:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25817088 chunk: [25817088, 25821184) 25821184, 4096 00020000:00200000:0.0:1789760763.936373:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.936415:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6303, 6303] 00000008:00010000:0.0:1789760763.936435:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.936441:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.936452:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25817088, 25821184) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.936464:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25817088=0x189f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.936524:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.936527:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25821184 00000020:00010000:0.0:1789760763.936535:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.936538:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.936542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.936545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.936548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.936557:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.936563:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.936579:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760763.936679:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 25817088<>25821184 00000100:00100000:1.0:1789760763.937069:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.937093:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790976:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760763.980663:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973622 gen 3 00000008:00010000:1.0:1789760763.980726:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760763.980735:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.980747:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.980775:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760763.980807:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493790976:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760763.980980:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6303, 6303] 00000080:00200000:0.0:1789760763.980998:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.981004:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.981019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25821184 i_blocks 51207 00000020:00200000:0.0:1789760763.981023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.981034:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.981067:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25821184, count: 4096 00000080:00200000:0.0:1789760763.981075:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25821184, bytes: 4096 00000080:00200000:0.0:1789760763.981089:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.981098:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.981101:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25821184, 25825280) 0 0 00000080:00200000:0.0:1789760763.981105:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.981121:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6304, 6304] 00020000:00200000:0.0:1789760763.981130:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25821184 chunk: [25821184, 25825280) 25825280, 4096 00020000:00200000:0.0:1789760763.981135:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.981155:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6304, 6304] 00000008:00010000:0.0:1789760763.981186:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.981193:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760763.981206:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25821184, 25825280) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.981222:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25821184=0x18a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.981287:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.981290:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25825280 00000020:00010000:0.0:1789760763.981299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.981302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.981305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.981308:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.981311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.981319:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.981327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.981342:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760763.981484:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25821184<>25825280 00000100:00100000:1.0:1789760763.981688:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.981705:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791104:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760764.021224:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973623 gen 3 00000008:00010000:1.0:1789760764.021789:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760764.021798:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.021810:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.021836:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760764.021865:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791104:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.021994:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6304, 6304] 00000080:00200000:0.0:1789760764.022010:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.022016:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.022033:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25825280 i_blocks 51207 00000020:00200000:0.0:1789760764.022037:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.022048:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.022095:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25825280, count: 4096 00000080:00200000:0.0:1789760764.022103:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25825280, bytes: 4096 00000080:00200000:0.0:1789760764.022116:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.022125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.022128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25825280, 25829376) 0 0 00000080:00200000:0.0:1789760764.022133:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.022150:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6305, 6305] 00020000:00200000:0.0:1789760764.022159:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25825280 chunk: [25825280, 25829376) 25829376, 4096 00020000:00200000:0.0:1789760764.022173:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.022194:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6305, 6305] 00000008:00010000:0.0:1789760764.022211:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.022217:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.022228:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25825280, 25829376) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.022243:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25825280=0x18a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.022743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.022756:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25829376 00000020:00010000:0.0:1789760764.022768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.022772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.022776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.022779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.022782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.022793:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.022800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.022824:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760764.022909:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 25825280<>25829376 00000100:00100000:1.0:1789760764.022992:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.023019:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791232:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760764.059506:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760764.059522:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.059533:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.059565:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760764.059593:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791232:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.059722:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6305, 6305] 00000080:00200000:0.0:1789760764.059738:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.059745:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.059760:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25829376 i_blocks 51207 00000020:00200000:0.0:1789760764.059764:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.059774:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.059806:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25829376, count: 4096 00000080:00200000:0.0:1789760764.059813:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25829376, bytes: 4096 00000080:00200000:0.0:1789760764.059833:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.059841:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.059844:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25829376, 25833472) 0 0 00000080:00200000:0.0:1789760764.059848:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.059866:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6306, 6306] 00020000:00200000:0.0:1789760764.059874:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25829376 chunk: [25829376, 25833472) 25833472, 4096 00020000:00200000:0.0:1789760764.059946:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.059966:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6306, 6306] 00000008:00010000:0.0:1789760764.059984:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.059990:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.060001:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25829376, 25833472) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.060015:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25829376=0x18a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.060076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.060079:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25833472 00000020:00010000:0.0:1789760764.060087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.060090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.060093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.060096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.060099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.060107:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.060114:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.060131:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760764.060744:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 25829376<>25833472 00000100:00100000:1.0:1789760764.061301:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760764.061326:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791360:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760764.105910:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973625 gen 3 00000008:00010000:1.0:1789760764.105988:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760764.105998:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.106009:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.106101:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760764.106141:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791360:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.108881:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6306, 6306] 00000080:00200000:0.0:1789760764.108906:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.108913:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.108930:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25833472 i_blocks 51207 00000020:00200000:0.0:1789760764.108936:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.108952:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.109783:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25833472, count: 4096 00000080:00200000:0.0:1789760764.109797:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25833472, bytes: 4096 00000080:00200000:0.0:1789760764.109811:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.109819:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.109822:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25833472, 25837568) 0 0 00000080:00200000:0.0:1789760764.109827:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.109842:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6307, 6307] 00020000:00200000:0.0:1789760764.109851:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25833472 chunk: [25833472, 25837568) 25837568, 4096 00020000:00200000:0.0:1789760764.109856:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.109875:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6307, 6307] 00000008:00010000:0.0:1789760764.109892:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.109897:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.109908:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25833472, 25837568) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.109921:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25833472=0x18a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.110360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.110364:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25837568 00000020:00010000:0.0:1789760764.110374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.110378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.110381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.110385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.110388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.110431:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.110438:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.110461:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760764.110565:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 25833472<>25837568 00000100:00100000:1.0:1789760764.111114:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791488:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760764.141209:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760764.168919:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760764.168936:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.168948:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.168970:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760764.169001:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493791488:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.169133:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6307, 6307] 00000080:00200000:0.0:1789760764.169149:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.169155:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.169178:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25837568 i_blocks 51207 00000020:00200000:0.0:1789760764.169182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.169193:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.169237:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25837568, count: 4096 00000080:00200000:0.0:1789760764.169243:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25837568, bytes: 4096 00000080:00200000:0.0:1789760764.169253:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.169261:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.169263:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25837568, 25841664) 0 0 00000080:00200000:0.0:1789760764.169267:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.169285:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6308, 6308] 00020000:00200000:0.0:1789760764.169293:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25837568 chunk: [25837568, 25841664) 25841664, 4096 00020000:00200000:0.0:1789760764.169297:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.169316:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6308, 6308] 00000008:00010000:0.0:1789760764.169334:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.169340:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.169352:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25837568, 25841664) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.169366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25837568=0x18a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.169458:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.169461:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25841664 00000020:00010000:0.0:1789760764.169469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.169473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.169476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.169480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.169483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.169491:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.169498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.169514:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760764.169807:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 25837568<>25841664 00000100:00100000:1.0:1789760764.169972:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791616:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760764.232478:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973627 gen 3 00000008:00010000:1.0:1789760764.232547:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760764.232556:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.232566:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.232635:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760764.232670:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791616:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.233669:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6308, 6308] 00000080:00200000:0.0:1789760764.233692:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.233699:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.233715:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25841664 i_blocks 51207 00000020:00200000:0.0:1789760764.233718:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.233731:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.233766:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25841664, count: 4096 00000080:00200000:0.0:1789760764.233773:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25841664, bytes: 4096 00000080:00200000:0.0:1789760764.233786:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.233794:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.233796:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25841664, 25845760) 0 0 00000080:00200000:0.0:1789760764.233801:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.233903:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6309, 6309] 00020000:00200000:0.0:1789760764.233913:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25841664 chunk: [25841664, 25845760) 25845760, 4096 00020000:00200000:0.0:1789760764.233918:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.233938:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6309, 6309] 00000008:00010000:0.0:1789760764.233956:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.233962:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.233974:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25841664, 25845760) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.233989:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25841664=0x18a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.234050:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.234181:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25845760 00000020:00010000:0.0:1789760764.234191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.234195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.234198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.234202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.234205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.234215:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.234222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.234248:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760764.234486:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 25841664<>25845760 00000100:00100000:1.0:1789760764.234718:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.234744:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791744:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760764.296635:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973628 gen 3 00000008:00010000:1.0:1789760764.296687:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760764.296696:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.296708:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.296789:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760764.297766:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493791744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760764.299466:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6309, 6309] 00000080:00200000:2.0:1789760764.299493:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.299501:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.299520:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25845760 i_blocks 51207 00000020:00200000:2.0:1789760764.299526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.299544:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.299577:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25845760, count: 4096 00000080:00200000:2.0:1789760764.299585:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25845760, bytes: 4096 00000080:00200000:2.0:1789760764.299601:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.299610:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.299613:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25845760, 25849856) 0 0 00000080:00200000:2.0:1789760764.299617:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.299634:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6310, 6310] 00020000:00200000:2.0:1789760764.299643:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25845760 chunk: [25845760, 25849856) 25849856, 4096 00020000:00200000:2.0:1789760764.299648:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.299670:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6310, 6310] 00000008:00010000:2.0:1789760764.299688:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.299693:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760764.299706:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25845760, 25849856) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.299721:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25845760=0x18a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.299791:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.299794:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25849856 00000020:00010000:2.0:1789760764.299802:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.299806:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.299808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.299812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.299826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.299836:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.299844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.299867:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760764.299973:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 25845760<>25849856 00000100:00100000:3.0:1789760764.300123:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.300152:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493791872:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.359854:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973629 gen 3 00000008:00010000:2.0:1789760764.359955:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760764.359963:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.359976:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.359998:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760764.360025:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493791872:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760764.360498:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6310, 6310] 00000080:00200000:1.0:1789760764.360514:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760764.360520:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760764.360534:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25849856 i_blocks 51207 00000020:00200000:1.0:1789760764.360538:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760764.360549:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760764.360579:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25849856, count: 4096 00000080:00200000:1.0:1789760764.360585:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25849856, bytes: 4096 00000080:00200000:1.0:1789760764.360597:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760764.360605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760764.360607:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25849856, 25853952) 0 0 00000080:00200000:1.0:1789760764.360611:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760764.360626:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6311, 6311] 00020000:00200000:1.0:1789760764.360634:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25849856 chunk: [25849856, 25853952) 25853952, 4096 00020000:00200000:1.0:1789760764.360638:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760764.360658:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6311, 6311] 00000008:00010000:1.0:1789760764.360734:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760764.360740:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760764.360752:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25849856, 25853952) append 0 append lockpos 0 00000080:00200000:1.0:1789760764.360765:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25849856=0x18a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760764.360824:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760764.360827:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25853952 00000020:00010000:1.0:1789760764.360836:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760764.360840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.360843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760764.360847:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760764.360849:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760764.360860:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760764.360867:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760764.360935:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760764.361035:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 25849856<>25853952 00000100:00100000:0.0:1789760764.361213:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493792000:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760764.393667:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973630 gen 3 00000008:00010000:0.0:1789760764.393732:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760764.393741:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760764.393752:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760764.393775:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760764.393804:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493792000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760764.393935:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6311, 6311] 00000080:00200000:2.0:1789760764.393952:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.393957:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.393985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25853952 i_blocks 51207 00000020:00200000:2.0:1789760764.393988:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.393999:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.394037:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25853952, count: 4096 00000080:00200000:2.0:1789760764.394044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25853952, bytes: 4096 00000080:00200000:2.0:1789760764.394054:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.394062:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.394065:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25853952, 25858048) 0 0 00000080:00200000:2.0:1789760764.394070:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.394085:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6312, 6312] 00020000:00200000:2.0:1789760764.394093:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25853952 chunk: [25853952, 25858048) 25858048, 4096 00020000:00200000:2.0:1789760764.394104:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.394127:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6312, 6312] 00000008:00010000:2.0:1789760764.394147:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.394153:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760764.394176:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25853952, 25858048) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.394191:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25853952=0x18a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.394375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.394380:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25858048 00000020:00010000:2.0:1789760764.394551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.394556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.394560:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.394563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.394567:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.394576:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.394584:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.394602:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760764.394727:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 25853952<>25858048 00000100:00100000:2.0:1789760764.394779:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792128:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760764.432096:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760764.432113:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.432126:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.432146:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:2.0:1789760764.432183:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792128:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.432365:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6312, 6312] 00000080:00200000:3.0:1789760764.432382:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.432388:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.432432:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25858048 i_blocks 51207 00000020:00200000:3.0:1789760764.432436:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.432447:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.432480:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25858048, count: 4096 00000080:00200000:3.0:1789760764.432487:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25858048, bytes: 4096 00000080:00200000:3.0:1789760764.432498:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.432507:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.432509:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25858048, 25862144) 0 0 00000080:00200000:3.0:1789760764.432513:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.432531:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6313, 6313] 00020000:00200000:3.0:1789760764.432538:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25858048 chunk: [25858048, 25862144) 25862144, 4096 00020000:00200000:3.0:1789760764.432543:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.432563:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6313, 6313] 00000008:00010000:3.0:1789760764.432580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.432586:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.432598:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25858048, 25862144) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.432612:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25858048=0x18a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.432678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.432681:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25862144 00000020:00010000:3.0:1789760764.432689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.432692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.432695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.432699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.432701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.432709:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.432717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.432733:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760764.432846:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25858048<>25862144 00000100:00100000:2.0:1789760764.433051:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792256:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.462601:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973631 gen 3 00000100:00100000:2.0:1789760764.462641:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b100 x1876699493792256/t4294973632(4294973632) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760780 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760764.462690:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760764.462700:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.462712:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.462733:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760764.462756:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792256:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.462920:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6313, 6313] 00000080:00200000:3.0:1789760764.462935:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.462940:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.462956:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25862144 i_blocks 51207 00000020:00200000:3.0:1789760764.462959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.462968:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.463000:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25862144, count: 4096 00000080:00200000:3.0:1789760764.463006:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25862144, bytes: 4096 00000080:00200000:3.0:1789760764.463069:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.463082:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.463085:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25862144, 25866240) 0 0 00000080:00200000:3.0:1789760764.463091:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.463117:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6314, 6314] 00020000:00200000:3.0:1789760764.463132:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25862144 chunk: [25862144, 25866240) 25866240, 4096 00020000:00200000:3.0:1789760764.463140:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.463175:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6314, 6314] 00000008:00010000:3.0:1789760764.463201:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.463209:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.463225:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25862144, 25866240) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.463239:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25862144=0x18aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.463310:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.463313:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25866240 00000020:00010000:3.0:1789760764.463329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.463333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.463339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.463345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.463350:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.463358:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.463367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.463381:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760764.463894:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25862144<>25866240 00000100:00100000:3.0:1789760764.464006:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760764.464027:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792384:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.515234:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973633 gen 3 00000008:00010000:2.0:1789760764.515301:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760764.515311:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.515321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.515434:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760764.515464:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792384:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760764.517129:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6314, 6314] 00000080:00200000:0.0:1789760764.517152:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.517168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.517188:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25866240 i_blocks 51207 00000020:00200000:0.0:1789760764.517194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.517207:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.517927:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25866240, count: 4096 00000080:00200000:0.0:1789760764.517940:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25866240, bytes: 4096 00000080:00200000:0.0:1789760764.517955:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.517964:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.517966:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25866240, 25870336) 0 0 00000080:00200000:0.0:1789760764.517972:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.517988:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6315, 6315] 00020000:00200000:0.0:1789760764.517998:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25866240 chunk: [25866240, 25870336) 25870336, 4096 00020000:00200000:0.0:1789760764.518004:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.518025:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6315, 6315] 00000008:00010000:0.0:1789760764.518044:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.518051:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760764.518063:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25866240, 25870336) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.518079:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25866240=0x18ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.518150:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.518154:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25870336 00000020:00010000:0.0:1789760764.518171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.518175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.518178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.518181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.518184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.518192:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.518199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.518560:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760764.518671:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 25866240<>25870336 00000100:00100000:1.0:1789760764.518753:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493792512:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760764.584259:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973634 gen 3 00000008:00010000:1.0:1789760764.584315:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760764.584324:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.584335:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.584546:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760764.584576:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493792512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760764.584799:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6315, 6315] 00000080:00200000:2.0:1789760764.584817:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.584824:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.584839:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25870336 i_blocks 51207 00000020:00200000:2.0:1789760764.584844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.584856:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.584891:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25870336, count: 4096 00000080:00200000:2.0:1789760764.584899:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25870336, bytes: 4096 00000080:00200000:2.0:1789760764.584910:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.584920:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.584922:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25870336, 25874432) 0 0 00000080:00200000:2.0:1789760764.584927:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.584943:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6316, 6316] 00020000:00200000:2.0:1789760764.584954:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25870336 chunk: [25870336, 25874432) 25874432, 4096 00020000:00200000:2.0:1789760764.584958:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.584980:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6316, 6316] 00000008:00010000:2.0:1789760764.584998:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.585004:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760764.585016:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25870336, 25874432) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.585030:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25870336=0x18ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.585253:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.585257:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25874432 00000020:00010000:2.0:1789760764.585266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.585270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.585273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.585277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.585280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.585289:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.585295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.585312:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760764.587015:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 25870336<>25874432 00000100:00100000:2.0:1789760764.587151:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760764.587191:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792640:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.613490:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973635 gen 3 00000008:00010000:2.0:1789760764.613539:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:2.0:1789760764.613546:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.613555:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.613571:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000080:00200000:3.0:1789760764.613596:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6316, 6316] 00000100:00100000:2.0:1789760764.613600:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792640:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.613612:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.613618:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.613633:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25874432 i_blocks 51207 00000020:00200000:3.0:1789760764.613637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.613647:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.613681:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25874432, count: 4096 00000080:00200000:3.0:1789760764.613689:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25874432, bytes: 4096 00000080:00200000:3.0:1789760764.613698:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.613708:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.613711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25874432, 25878528) 0 0 00000080:00200000:3.0:1789760764.613715:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.613733:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6317, 6317] 00020000:00200000:3.0:1789760764.613741:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25874432 chunk: [25874432, 25878528) 25878528, 4096 00020000:00200000:3.0:1789760764.613744:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.613761:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6317, 6317] 00000008:00010000:3.0:1789760764.613774:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.613779:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.613843:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25874432, 25878528) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.613857:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25874432=0x18ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.613915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.613919:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25878528 00000020:00010000:3.0:1789760764.613927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.613931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.613934:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.613937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.613940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.613948:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.614016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.614033:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760764.614130:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25874432<>25878528 00000100:00100000:2.0:1789760764.614204:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792768:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.644713:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973636 gen 3 00000008:00010000:2.0:1789760764.644762:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760764.644770:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.644782:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.644804:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760764.644834:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493792768:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.645002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6317, 6317] 00000080:00200000:3.0:1789760764.645019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.645025:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.645040:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25878528 i_blocks 51207 00000020:00200000:3.0:1789760764.645044:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.645053:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.645086:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25878528, count: 4096 00000080:00200000:3.0:1789760764.645093:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25878528, bytes: 4096 00000080:00200000:3.0:1789760764.645103:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.645112:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.645115:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25878528, 25882624) 0 0 00000080:00200000:3.0:1789760764.645120:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.645135:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6318, 6318] 00020000:00200000:3.0:1789760764.645374:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25878528 chunk: [25878528, 25882624) 25882624, 4096 00020000:00200000:3.0:1789760764.645380:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.645490:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6318, 6318] 00000008:00010000:3.0:1789760764.645510:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.645516:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.645528:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25878528, 25882624) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.645541:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25878528=0x18ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.645601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.645604:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25882624 00000020:00010000:3.0:1789760764.645612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.645616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.645619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.645623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.645626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.645634:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.645641:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.645658:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760764.646049:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 25878528<>25882624 00000100:00100000:2.0:1789760764.646301:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792896:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.690428:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973637 gen 3 00000008:00010000:2.0:1789760764.690486:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760764.690495:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.690507:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.690526:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760764.690559:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493792896:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.690998:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6318, 6318] 00000080:00200000:3.0:1789760764.691018:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.691029:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.691048:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25882624 i_blocks 51207 00000020:00200000:3.0:1789760764.691051:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.691062:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.691097:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25882624, count: 4096 00000080:00200000:3.0:1789760764.691105:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25882624, bytes: 4096 00000080:00200000:3.0:1789760764.691119:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.691128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.691131:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25882624, 25886720) 0 0 00000080:00200000:3.0:1789760764.691143:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.691168:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6319, 6319] 00020000:00200000:3.0:1789760764.691178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25882624 chunk: [25882624, 25886720) 25886720, 4096 00020000:00200000:3.0:1789760764.691183:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.691208:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6319, 6319] 00000008:00010000:3.0:1789760764.691227:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.691237:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.691249:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25882624, 25886720) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.691263:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25882624=0x18af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.691331:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.691333:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25886720 00000020:00010000:3.0:1789760764.691342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.691346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.691352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.691356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.691359:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.691368:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.691375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.691432:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760764.691557:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 25882624<>25886720 00000100:00100000:2.0:1789760764.691758:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760764.691776:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793024:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.732718:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973638 gen 3 00000008:00010000:2.0:1789760764.732772:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760764.732780:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.732793:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.732812:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760764.732854:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793024:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.733310:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6319, 6319] 00000080:00200000:3.0:1789760764.733328:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.733335:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.733351:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25886720 i_blocks 51207 00000020:00200000:3.0:1789760764.733355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.733365:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.733421:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25886720, count: 4096 00000080:00200000:3.0:1789760764.733429:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25886720, bytes: 4096 00000080:00200000:3.0:1789760764.733441:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.733450:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.733452:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25886720, 25890816) 0 0 00000080:00200000:3.0:1789760764.733457:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.733474:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6320, 6320] 00020000:00200000:3.0:1789760764.733482:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25886720 chunk: [25886720, 25890816) 25890816, 4096 00020000:00200000:3.0:1789760764.733487:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.733507:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6320, 6320] 00000008:00010000:3.0:1789760764.733527:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.733533:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.733546:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25886720, 25890816) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.733560:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25886720=0x18b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.733621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.733625:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25890816 00000020:00010000:3.0:1789760764.733635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.733708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.733712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.733715:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.733718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.733727:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.733734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.733751:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760764.733874:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 25886720<>25890816 00000100:00100000:2.0:1789760764.734341:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793152:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.785785:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973639 gen 3 00000008:00010000:2.0:1789760764.785848:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760764.785856:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.785867:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.785895:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760764.785927:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793152:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.786421:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6320, 6320] 00000080:00200000:3.0:1789760764.786438:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.786445:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.786460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25890816 i_blocks 51207 00000020:00200000:3.0:1789760764.786464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.786474:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.786511:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25890816, count: 4096 00000080:00200000:3.0:1789760764.786519:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25890816, bytes: 4096 00000080:00200000:3.0:1789760764.786529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.786537:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.786540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25890816, 25894912) 0 0 00000080:00200000:3.0:1789760764.786545:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.786560:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6321, 6321] 00020000:00200000:3.0:1789760764.786568:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25890816 chunk: [25890816, 25894912) 25894912, 4096 00020000:00200000:3.0:1789760764.786573:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.786591:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6321, 6321] 00000008:00010000:3.0:1789760764.786609:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.786615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.786627:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25890816, 25894912) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.786710:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25890816=0x18b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.786770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.786773:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25894912 00000020:00010000:3.0:1789760764.786783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.786786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.786789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.786793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.786796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.786804:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.786812:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.786841:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760764.787150:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 25890816<>25894912 00000100:00100000:2.0:1789760764.787350:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793280:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760764.830554:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760764.830571:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.830583:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.830604:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760764.830635:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793280:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.830731:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6321, 6321] 00000080:00200000:3.0:1789760764.830748:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.830754:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.830769:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25894912 i_blocks 51207 00000020:00200000:3.0:1789760764.830773:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.830783:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.830815:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25894912, count: 4096 00000080:00200000:3.0:1789760764.830822:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25894912, bytes: 4096 00000080:00200000:3.0:1789760764.830830:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.830840:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.830842:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25894912, 25899008) 0 0 00000080:00200000:3.0:1789760764.830846:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.830861:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6322, 6322] 00020000:00200000:3.0:1789760764.830869:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25894912 chunk: [25894912, 25899008) 25899008, 4096 00020000:00200000:3.0:1789760764.830874:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.830892:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6322, 6322] 00000008:00010000:3.0:1789760764.830910:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.830916:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.830929:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25894912, 25899008) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.830952:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25894912=0x18b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.831016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.831019:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25899008 00000020:00010000:3.0:1789760764.831029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.831032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.831035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.831039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.831042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.831050:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.831056:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.831073:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760764.831186:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 25894912<>25899008 00000100:00100000:2.0:1789760764.831522:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793408:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.874194:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973641 gen 3 00000008:00010000:2.0:1789760764.874326:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760764.874335:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.874345:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.874372:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760764.874434:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793408:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.877561:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6322, 6322] 00000080:00200000:3.0:1789760764.877592:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.877599:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.877617:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25899008 i_blocks 51207 00000020:00200000:3.0:1789760764.877623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.877636:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.877673:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25899008, count: 4096 00000080:00200000:3.0:1789760764.877681:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25899008, bytes: 4096 00000080:00200000:3.0:1789760764.877692:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.877702:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.877705:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25899008, 25903104) 0 0 00000080:00200000:3.0:1789760764.877710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.877727:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6323, 6323] 00020000:00200000:3.0:1789760764.877735:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25899008 chunk: [25899008, 25903104) 25903104, 4096 00020000:00200000:3.0:1789760764.877740:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.877761:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6323, 6323] 00000008:00010000:3.0:1789760764.877781:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.877787:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.878013:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25899008, 25903104) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.878031:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25899008=0x18b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.878717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.878723:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25903104 00000020:00010000:3.0:1789760764.878735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.878739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.878741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.878745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.878748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.878757:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.878766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.878792:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760764.878995:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 25899008<>25903104 00000100:00100000:2.0:1789760764.879109:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793536:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.937329:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973642 gen 3 00000008:00010000:2.0:1789760764.937382:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760764.937391:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.937439:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.937462:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760764.937489:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793536:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.937586:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6323, 6323] 00000080:00200000:3.0:1789760764.937603:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.937609:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.937624:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25903104 i_blocks 51207 00000020:00200000:3.0:1789760764.937629:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.937713:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.937752:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25903104, count: 4096 00000080:00200000:3.0:1789760764.937760:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25903104, bytes: 4096 00000080:00200000:3.0:1789760764.937770:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.937780:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.937782:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25903104, 25907200) 0 0 00000080:00200000:3.0:1789760764.937787:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.937804:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6324, 6324] 00020000:00200000:3.0:1789760764.937812:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25903104 chunk: [25903104, 25907200) 25907200, 4096 00020000:00200000:3.0:1789760764.937817:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.937836:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6324, 6324] 00000008:00010000:3.0:1789760764.937855:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.937861:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.937873:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25903104, 25907200) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.937888:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25903104=0x18b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.937950:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.937953:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25907200 00000020:00010000:3.0:1789760764.937962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.937966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.937969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.937972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.937975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.937984:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.937991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.938007:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760764.938334:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 25903104<>25907200 00000100:00100000:2.0:1789760764.938468:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793664:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760764.990237:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973643 gen 3 00000008:00010000:2.0:1789760764.990286:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760764.990295:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760764.990306:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760764.990326:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760764.990354:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793664:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760764.990467:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6324, 6324] 00000080:00200000:3.0:1789760764.990483:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760764.990489:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760764.990503:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25907200 i_blocks 51207 00000020:00200000:3.0:1789760764.990506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760764.990515:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760764.990545:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25907200, count: 4096 00000080:00200000:3.0:1789760764.990551:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25907200, bytes: 4096 00000080:00200000:3.0:1789760764.990560:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760764.990569:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760764.990572:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25907200, 25911296) 0 0 00000080:00200000:3.0:1789760764.990577:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760764.990593:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6325, 6325] 00020000:00200000:3.0:1789760764.990601:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25907200 chunk: [25907200, 25911296) 25911296, 4096 00020000:00200000:3.0:1789760764.990605:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760764.990624:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6325, 6325] 00000008:00010000:3.0:1789760764.990651:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760764.990657:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760764.990668:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25907200, 25911296) append 0 append lockpos 0 00000080:00200000:3.0:1789760764.990681:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25907200=0x18b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760764.990733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760764.990736:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25911296 00000020:00010000:3.0:1789760764.990743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760764.990747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.990750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760764.990754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760764.990757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760764.990763:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760764.990769:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760764.990783:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760764.990878:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 25907200<>25911296 00000100:00100000:2.0:1789760764.991028:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793792:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.082656:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973644 gen 3 00000008:00010000:2.0:1789760765.082704:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760765.082713:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.082724:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.082741:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760765.082772:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493793792:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.083030:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6325, 6325] 00000080:00200000:3.0:1789760765.083056:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.083062:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.083078:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25911296 i_blocks 51207 00000020:00200000:3.0:1789760765.083082:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.083092:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.083125:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25911296, count: 4096 00000080:00200000:3.0:1789760765.083133:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25911296, bytes: 4096 00000080:00200000:3.0:1789760765.083152:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.083170:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.083173:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25911296, 25915392) 0 0 00000080:00200000:3.0:1789760765.083177:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.083195:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6326, 6326] 00020000:00200000:3.0:1789760765.083204:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25911296 chunk: [25911296, 25915392) 25915392, 4096 00020000:00200000:3.0:1789760765.083208:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.083227:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6326, 6326] 00000008:00010000:3.0:1789760765.083245:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.083251:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.083263:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25911296, 25915392) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.083279:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25911296=0x18b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.083339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.083342:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25915392 00000020:00010000:3.0:1789760765.083351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.083354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.083357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.083361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.083363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.083371:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.083378:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.083429:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760765.083527:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 25911296<>25915392 00000100:00100000:2.0:1789760765.083680:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793920:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760765.165116:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760765.180851:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973645 gen 3 00000008:00010000:2.0:1789760765.180902:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760765.180911:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.180933:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.180951:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760765.180982:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493793920:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.181143:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6326, 6326] 00000080:00200000:3.0:1789760765.181167:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.181173:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.181190:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25915392 i_blocks 51207 00000020:00200000:3.0:1789760765.181194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.181204:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.181240:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25915392, count: 4096 00000080:00200000:3.0:1789760765.181247:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25915392, bytes: 4096 00000080:00200000:3.0:1789760765.181257:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.181266:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.181268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25915392, 25919488) 0 0 00000080:00200000:3.0:1789760765.181273:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.181290:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6327, 6327] 00020000:00200000:3.0:1789760765.181299:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25915392 chunk: [25915392, 25919488) 25919488, 4096 00020000:00200000:3.0:1789760765.181303:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.181323:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6327, 6327] 00000008:00010000:3.0:1789760765.181342:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.181348:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.181359:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25915392, 25919488) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.181374:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25915392=0x18b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.181529:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.181598:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25919488 00000020:00010000:3.0:1789760765.181607:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.181611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.181614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.181618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.181621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.181629:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.181700:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.181718:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760765.181824:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 25915392<>25919488 00000100:00100000:2.0:1789760765.181894:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794048:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.247612:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973646 gen 3 00000008:00010000:2.0:1789760765.247667:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760765.247676:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.247686:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.247706:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760765.247735:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794048:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.248215:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6327, 6327] 00000080:00200000:3.0:1789760765.248232:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.248239:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.248256:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25919488 i_blocks 51207 00000020:00200000:3.0:1789760765.248261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.248271:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.253384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25919488, count: 4096 00000080:00200000:3.0:1789760765.253426:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25919488, bytes: 4096 00000080:00200000:3.0:1789760765.253448:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.253460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.253462:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25919488, 25923584) 0 0 00000080:00200000:3.0:1789760765.253469:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.253490:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6328, 6328] 00020000:00200000:3.0:1789760765.253500:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25919488 chunk: [25919488, 25923584) 25923584, 4096 00020000:00200000:3.0:1789760765.253504:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.253527:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6328, 6328] 00000008:00010000:3.0:1789760765.253550:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.253557:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.253569:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25919488, 25923584) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.253582:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25919488=0x18b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.253669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.253673:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25923584 00000020:00010000:3.0:1789760765.253682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.253685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.253689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.253692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.253695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.253707:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.253715:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.253753:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760765.253867:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 25919488<>25923584 00000100:00100000:2.0:1789760765.255657:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794176:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760765.337976:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760765.337994:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.338006:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.338028:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760765.338059:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794176:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.340804:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6328, 6328] 00000080:00200000:3.0:1789760765.340841:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.340848:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.340868:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25923584 i_blocks 51207 00000020:00200000:3.0:1789760765.340873:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.340891:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.341027:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25923584, count: 4096 00000080:00200000:3.0:1789760765.341037:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25923584, bytes: 4096 00000080:00200000:3.0:1789760765.341051:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.341060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.341062:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25923584, 25927680) 0 0 00000080:00200000:3.0:1789760765.341067:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.341083:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6329, 6329] 00020000:00200000:3.0:1789760765.341091:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25923584 chunk: [25923584, 25927680) 25927680, 4096 00020000:00200000:3.0:1789760765.341096:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.341116:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6329, 6329] 00000008:00010000:3.0:1789760765.341134:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.341139:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.341150:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25923584, 25927680) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.341176:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25923584=0x18b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.341251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.341254:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25927680 00000020:00010000:3.0:1789760765.341262:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.341266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.341268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.341272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.341275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.341284:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.341290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.341312:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760765.341463:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 25923584<>25927680 00000100:00100000:2.0:1789760765.342007:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760765.342034:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794304:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.399615:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973647 gen 3 00000100:00100000:2.0:1789760765.399718:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148000 x1876699493794304/t4294973648(4294973648) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760781 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760765.399771:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760765.399778:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.399789:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.399847:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760765.399872:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794304:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.400441:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6329, 6329] 00000080:00200000:3.0:1789760765.400460:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.400466:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.400482:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25927680 i_blocks 51207 00000020:00200000:3.0:1789760765.400485:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.400496:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.400534:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25927680, count: 4096 00000080:00200000:3.0:1789760765.400542:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25927680, bytes: 4096 00000080:00200000:3.0:1789760765.400551:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.400560:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.400562:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25927680, 25931776) 0 0 00000080:00200000:3.0:1789760765.400567:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.400583:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6330, 6330] 00020000:00200000:3.0:1789760765.400591:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25927680 chunk: [25927680, 25931776) 25931776, 4096 00020000:00200000:3.0:1789760765.400595:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.400614:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6330, 6330] 00000008:00010000:3.0:1789760765.400631:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.400637:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.400649:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25927680, 25931776) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.400662:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25927680=0x18ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.400721:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.400724:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25931776 00000020:00010000:3.0:1789760765.400732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.400736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.400739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.400742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.400745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.400754:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.400761:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.400778:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760765.401034:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 25927680<>25931776 00000100:00100000:2.0:1789760765.401147:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760765.401176:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794432:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.430465:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973649 gen 3 00000008:00010000:2.0:1789760765.430520:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760765.430528:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.430538:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.430559:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760765.430588:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794432:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.430856:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6330, 6330] 00000080:00200000:3.0:1789760765.430872:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.430878:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.430958:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25931776 i_blocks 51207 00000020:00200000:3.0:1789760765.430962:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.430971:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.431007:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25931776, count: 4096 00000080:00200000:3.0:1789760765.431015:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25931776, bytes: 4096 00000080:00200000:3.0:1789760765.431025:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.431035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.431037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25931776, 25935872) 0 0 00000080:00200000:3.0:1789760765.431042:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.431058:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6331, 6331] 00020000:00200000:3.0:1789760765.431066:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25931776 chunk: [25931776, 25935872) 25935872, 4096 00020000:00200000:3.0:1789760765.431071:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.431091:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6331, 6331] 00000008:00010000:3.0:1789760765.431108:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.431115:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.431125:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25931776, 25935872) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.431219:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25931776=0x18bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.431281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.431284:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25935872 00000020:00010000:3.0:1789760765.431293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.431297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.431300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.431303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.431306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.431313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.431321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.431336:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760765.431478:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 25931776<>25935872 00000100:00100000:2.0:1789760765.431725:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760765.431747:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794560:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.476484:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973650 gen 3 00000008:00010000:2.0:1789760765.476600:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760765.476609:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.476622:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.476634:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760765.476665:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794560:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.477469:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6331, 6331] 00000080:00200000:3.0:1789760765.477486:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.477491:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.477507:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25935872 i_blocks 51207 00000020:00200000:3.0:1789760765.477511:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.477521:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.477614:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25935872, count: 4096 00000080:00200000:3.0:1789760765.477623:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25935872, bytes: 4096 00000080:00200000:3.0:1789760765.477700:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.477711:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.477714:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25935872, 25939968) 0 0 00000080:00200000:3.0:1789760765.477718:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.477791:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6332, 6332] 00020000:00200000:3.0:1789760765.477801:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25935872 chunk: [25935872, 25939968) 25939968, 4096 00020000:00200000:3.0:1789760765.478000:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.478024:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6332, 6332] 00000008:00010000:3.0:1789760765.478044:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.478050:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.478061:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25935872, 25939968) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.478076:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25935872=0x18bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.478139:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.478142:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25939968 00000020:00010000:3.0:1789760765.478159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.478163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.478166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.478169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.478172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.478180:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.478187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.478272:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760765.478381:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 25935872<>25939968 00000100:00100000:2.0:1789760765.478539:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760765.478563:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794688:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.517541:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973651 gen 3 00000008:00010000:2.0:1789760765.517594:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760765.517601:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.517611:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.517624:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760765.517647:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493794688:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.517667:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6332, 6332] 00000080:00200000:3.0:1789760765.517687:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.517693:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.517709:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25939968 i_blocks 51207 00000020:00200000:3.0:1789760765.517713:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.517723:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.517759:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25939968, count: 4096 00000080:00200000:3.0:1789760765.517768:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25939968, bytes: 4096 00000080:00200000:3.0:1789760765.517778:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.517789:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.517791:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25939968, 25944064) 0 0 00000080:00200000:3.0:1789760765.517796:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.517825:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6333, 6333] 00020000:00200000:3.0:1789760765.517834:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25939968 chunk: [25939968, 25944064) 25944064, 4096 00020000:00200000:3.0:1789760765.517838:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.517859:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6333, 6333] 00000008:00010000:3.0:1789760765.517878:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.517884:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.517897:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25939968, 25944064) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.517912:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25939968=0x18bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.517971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.517974:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25944064 00000020:00010000:3.0:1789760765.517985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.517988:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.517992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.517995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.517998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.518006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.518013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.518030:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760765.518132:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 25939968<>25944064 00000100:00100000:2.0:1789760765.518610:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760765.518631:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794816:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760765.548058:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973652 gen 3 00000008:00010000:2.0:1789760765.548115:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760765.548126:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.548140:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.548236:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760765.548270:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794816:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.548437:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6333, 6333] 00000080:00200000:3.0:1789760765.548454:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.548461:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.548479:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25944064 i_blocks 51207 00000020:00200000:3.0:1789760765.548483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.548494:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.548530:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25944064, count: 4096 00000080:00200000:3.0:1789760765.548538:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25944064, bytes: 4096 00000080:00200000:3.0:1789760765.549464:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.549482:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.549485:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25944064, 25948160) 0 0 00000080:00200000:3.0:1789760765.549492:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.549510:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6334, 6334] 00020000:00200000:3.0:1789760765.549519:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25944064 chunk: [25944064, 25948160) 25948160, 4096 00020000:00200000:3.0:1789760765.549524:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.549556:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6334, 6334] 00000008:00010000:3.0:1789760765.549637:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.549645:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.549658:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25944064, 25948160) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.549674:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25944064=0x18be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.549737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.549740:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25948160 00000020:00010000:3.0:1789760765.549750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.549753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.549756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.549760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.549762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.549772:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.549780:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.549798:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760765.549906:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 25944064<>25948160 00000100:00100000:2.0:1789760765.549985:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794944:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760765.619973:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760765.619989:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.620001:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.620030:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760765.620060:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493794944:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.620450:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6334, 6334] 00000080:00200000:1.0:1789760765.620467:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.620555:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.620571:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25948160 i_blocks 51207 00000020:00200000:1.0:1789760765.620575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.620585:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.620618:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25948160, count: 4096 00000080:00200000:1.0:1789760765.620626:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25948160, bytes: 4096 00000080:00200000:1.0:1789760765.620639:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.620647:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.620649:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25948160, 25952256) 0 0 00000080:00200000:1.0:1789760765.620653:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.620740:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6335, 6335] 00020000:00200000:1.0:1789760765.620750:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25948160 chunk: [25948160, 25952256) 25952256, 4096 00020000:00200000:1.0:1789760765.620755:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.620775:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6335, 6335] 00000008:00010000:1.0:1789760765.620853:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.620859:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.620871:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25948160, 25952256) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.620885:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25948160=0x18bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.620945:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.620947:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25952256 00000020:00010000:1.0:1789760765.620955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.620958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.620962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.620965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.620968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.620979:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.620985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.621001:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760765.621101:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 25948160<>25952256 00000100:00100000:0.0:1789760765.621256:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795072:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760765.659754:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973654 gen 3 00000008:00010000:0.0:1789760765.659838:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760765.659847:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.659857:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.659879:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760765.659909:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795072:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.660332:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6335, 6335] 00000080:00200000:1.0:1789760765.660349:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.660355:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.660370:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25952256 i_blocks 51207 00000020:00200000:1.0:1789760765.660373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.660383:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.660433:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25952256, count: 4096 00000080:00200000:1.0:1789760765.660440:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25952256, bytes: 4096 00000080:00200000:1.0:1789760765.660448:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.660457:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.660460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25952256, 25956352) 0 0 00000080:00200000:1.0:1789760765.660464:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.660480:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6336, 6336] 00020000:00200000:1.0:1789760765.660488:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25952256 chunk: [25952256, 25956352) 25956352, 4096 00020000:00200000:1.0:1789760765.660492:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.660511:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6336, 6336] 00000008:00010000:1.0:1789760765.660528:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.660534:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.660545:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25952256, 25956352) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.660559:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25952256=0x18c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.660616:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.660619:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25956352 00000020:00010000:1.0:1789760765.660627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.660631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.660634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.660637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.660640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.660650:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.660657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.660897:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760765.660996:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 25952256<>25956352 00000100:00100000:0.0:1789760765.661080:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795200:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760765.690828:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760765.690845:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.690857:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.690881:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760765.690913:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795200:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.691074:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6336, 6336] 00000080:00200000:1.0:1789760765.691090:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.691097:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.691111:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25956352 i_blocks 51207 00000020:00200000:1.0:1789760765.691115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.691125:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.691171:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25956352, count: 4096 00000080:00200000:1.0:1789760765.691178:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25956352, bytes: 4096 00000080:00200000:1.0:1789760765.691188:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.691196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.691198:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25956352, 25960448) 0 0 00000080:00200000:1.0:1789760765.691202:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.691218:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6337, 6337] 00020000:00200000:1.0:1789760765.691227:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25956352 chunk: [25956352, 25960448) 25960448, 4096 00020000:00200000:1.0:1789760765.691232:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.691251:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6337, 6337] 00000008:00010000:1.0:1789760765.691268:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.691274:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.691285:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25956352, 25960448) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.691299:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25956352=0x18c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.691356:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.691359:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25960448 00000020:00010000:1.0:1789760765.691369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.691372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.691376:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.691381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.691384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.691435:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.691443:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.691459:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760765.691561:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 25956352<>25960448 00000100:00100000:0.0:1789760765.694016:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795328:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760765.753782:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973656 gen 3 00000008:00010000:0.0:1789760765.754034:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760765.754042:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.754049:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.754068:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760765.754092:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795328:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.754547:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6337, 6337] 00000080:00200000:1.0:1789760765.754564:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.754570:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.754587:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25960448 i_blocks 51207 00000020:00200000:1.0:1789760765.754591:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.754602:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.754637:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25960448, count: 4096 00000080:00200000:1.0:1789760765.754644:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25960448, bytes: 4096 00000080:00200000:1.0:1789760765.754726:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.754735:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.754737:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25960448, 25964544) 0 0 00000080:00200000:1.0:1789760765.754742:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.754769:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6338, 6338] 00020000:00200000:1.0:1789760765.754778:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25960448 chunk: [25960448, 25964544) 25964544, 4096 00020000:00200000:1.0:1789760765.754783:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.754802:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6338, 6338] 00000008:00010000:1.0:1789760765.755142:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.755158:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.755172:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25960448, 25964544) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.755188:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25960448=0x18c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.755626:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.755630:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25964544 00000020:00010000:1.0:1789760765.755640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.755643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.755647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.756066:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.756071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.756087:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.756096:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.756115:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760765.756302:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 25960448<>25964544 00000100:00100000:0.0:1789760765.756384:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795456:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760765.794121:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973657 gen 3 00000008:00010000:0.0:1789760765.794624:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760765.794642:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.794660:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.794684:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:0.0:1789760765.794714:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795456:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.794954:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6338, 6338] 00000080:00200000:1.0:1789760765.794970:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.794976:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.794993:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25964544 i_blocks 51207 00000020:00200000:1.0:1789760765.794997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.795008:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.795042:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25964544, count: 4096 00000080:00200000:1.0:1789760765.795050:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25964544, bytes: 4096 00000080:00200000:1.0:1789760765.795075:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.795084:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.795086:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25964544, 25968640) 0 0 00000080:00200000:1.0:1789760765.795090:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.795107:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6339, 6339] 00020000:00200000:1.0:1789760765.795115:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25964544 chunk: [25964544, 25968640) 25968640, 4096 00020000:00200000:1.0:1789760765.795120:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.795139:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6339, 6339] 00000008:00010000:1.0:1789760765.795167:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.795173:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.795358:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25964544, 25968640) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.795373:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25964544=0x18c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.795456:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.795459:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25968640 00000020:00010000:1.0:1789760765.795468:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.795472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.795476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.795480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.795483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.795493:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.795500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.795516:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760765.795623:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 25964544<>25968640 00000100:00100000:0.0:1789760765.796582:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795584:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760765.842171:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760765.842198:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.842211:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.842297:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760765.842331:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795584:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760765.842543:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6339, 6339] 00000080:00200000:1.0:1789760765.842560:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.842566:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.842581:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25968640 i_blocks 51207 00000020:00200000:1.0:1789760765.842584:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.842595:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.842696:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25968640, count: 4096 00000080:00200000:1.0:1789760765.842824:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25968640, bytes: 4096 00000080:00200000:1.0:1789760765.842838:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.842847:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.842850:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25968640, 25972736) 0 0 00000080:00200000:1.0:1789760765.842855:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.842872:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6340, 6340] 00020000:00200000:1.0:1789760765.842881:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25968640 chunk: [25968640, 25972736) 25972736, 4096 00020000:00200000:1.0:1789760765.842885:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.842906:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6340, 6340] 00000008:00010000:1.0:1789760765.842924:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.842930:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760765.842941:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25968640, 25972736) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.842956:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25968640=0x18c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.843018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.843021:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25972736 00000020:00010000:1.0:1789760765.843030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.843033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.843037:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.843041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.843044:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.843055:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.843063:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.843081:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760765.845511:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 25968640<>25972736 00000100:00100000:0.0:1789760765.845601:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795712:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760765.890159:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973659 gen 3 00000008:00010000:0.0:1789760765.890297:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760765.890306:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.890316:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.890339:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760765.890371:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493795712:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760765.890745:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6340, 6340] 00000080:00200000:3.0:1789760765.890773:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.890779:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.890805:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25972736 i_blocks 51207 00000020:00200000:3.0:1789760765.890809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.890829:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.891219:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25972736, count: 4096 00000080:00200000:3.0:1789760765.891229:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25972736, bytes: 4096 00000080:00200000:3.0:1789760765.891243:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.891254:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.891256:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25972736, 25976832) 0 0 00000080:00200000:3.0:1789760765.891262:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.891280:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6341, 6341] 00020000:00200000:3.0:1789760765.891289:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25972736 chunk: [25972736, 25976832) 25976832, 4096 00020000:00200000:3.0:1789760765.891294:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.891316:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6341, 6341] 00000008:00010000:3.0:1789760765.891336:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.891343:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760765.891356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25972736, 25976832) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.891371:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25972736=0x18c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.891598:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.891602:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25976832 00000020:00010000:3.0:1789760765.891613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.891616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.891620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.891624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.891636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.891646:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.891654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.891676:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760765.891787:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25972736<>25976832 00000100:00100000:2.0:1789760765.892162:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493795840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760765.934004:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973660 gen 3 00000008:00010000:3.0:1789760765.934058:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760765.934066:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760765.934077:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760765.934513:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760765.934546:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493795840:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760765.934783:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6341, 6341] 00000080:00200000:0.0:1789760765.935326:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.935335:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.935355:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25976832 i_blocks 51207 00000020:00200000:0.0:1789760765.935360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.935373:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.935443:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25976832, count: 4096 00000080:00200000:0.0:1789760765.935451:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25976832, bytes: 4096 00000080:00200000:0.0:1789760765.935463:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.935473:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.935475:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25976832, 25980928) 0 0 00000080:00200000:0.0:1789760765.935480:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.935497:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6342, 6342] 00020000:00200000:0.0:1789760765.935506:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25976832 chunk: [25976832, 25980928) 25980928, 4096 00020000:00200000:0.0:1789760765.935511:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.935531:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6342, 6342] 00000008:00010000:0.0:1789760765.935551:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.935557:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760765.935570:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25976832, 25980928) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.935584:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25976832=0x18c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.935723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.935726:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25980928 00000020:00010000:0.0:1789760765.935736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.935740:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.935743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.935747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.935750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.935758:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.935766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.935849:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760765.936029:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 25976832<>25980928 00000100:00100000:1.0:1789760765.936123:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795968:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760765.984390:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973661 gen 3 00000008:00010000:1.0:1789760765.984481:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760765.984488:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.984498:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.984513:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760765.984542:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493795968:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760765.984619:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6342, 6342] 00000080:00200000:0.0:1789760765.984727:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.984733:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.984749:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25980928 i_blocks 51207 00000020:00200000:0.0:1789760765.984754:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.984765:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.984802:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25980928, count: 4096 00000080:00200000:0.0:1789760765.984809:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25980928, bytes: 4096 00000080:00200000:0.0:1789760765.984949:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.984960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.984962:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25980928, 25985024) 0 0 00000080:00200000:0.0:1789760765.984967:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.984983:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6343, 6343] 00020000:00200000:0.0:1789760765.984991:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25980928 chunk: [25980928, 25985024) 25985024, 4096 00020000:00200000:0.0:1789760765.984996:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.985017:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6343, 6343] 00000008:00010000:0.0:1789760765.985035:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.985041:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760765.985053:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25980928, 25985024) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.985067:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25980928=0x18c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.985125:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.985128:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25985024 00000020:00010000:0.0:1789760765.985136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.985140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.985143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.985146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.985162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.985169:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.985177:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.985194:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760765.985464:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 25980928<>25985024 00000100:00100000:1.0:1789760765.985551:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493796096:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760766.030248:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973662 gen 3 00000008:00010000:1.0:1789760766.030301:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760766.030310:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.030321:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.030345:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760766.030374:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493796096:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760766.030596:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6343, 6343] 00000080:00200000:0.0:1789760766.030613:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.030620:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.030961:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25985024 i_blocks 51207 00000020:00200000:0.0:1789760766.030968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.030979:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.031015:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25985024, count: 4096 00000080:00200000:0.0:1789760766.031023:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25985024, bytes: 4096 00000080:00200000:0.0:1789760766.031034:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.031042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.031044:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25985024, 25989120) 0 0 00000080:00200000:0.0:1789760766.031048:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.031063:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6344, 6344] 00020000:00200000:0.0:1789760766.031072:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25985024 chunk: [25985024, 25989120) 25989120, 4096 00020000:00200000:0.0:1789760766.031077:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.031096:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6344, 6344] 00000008:00010000:0.0:1789760766.031112:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.031118:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760766.031130:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25985024, 25989120) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.031257:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25985024=0x18c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.031321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.031324:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25989120 00000020:00010000:0.0:1789760766.031333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.031337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.031340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.031344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.031347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.031354:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.031362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.032814:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760766.033679:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 25985024<>25989120 00000100:00100000:0.0:1789760766.033754:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760766.033782:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493796224:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760766.091292:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973663 gen 3 00000008:00010000:0.0:1789760766.091343:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760766.091351:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.091362:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.091438:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760766.091459:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493796224:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760766.091601:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6344, 6344] 00000080:00200000:1.0:1789760766.091617:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.091624:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.091650:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25989120 i_blocks 51207 00000020:00200000:1.0:1789760766.091654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.091665:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.091699:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25989120, count: 4096 00000080:00200000:1.0:1789760766.091707:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25989120, bytes: 4096 00000080:00200000:1.0:1789760766.091726:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.091735:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.091738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25989120, 25993216) 0 0 00000080:00200000:1.0:1789760766.091743:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.091759:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6345, 6345] 00020000:00200000:1.0:1789760766.091769:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25989120 chunk: [25989120, 25993216) 25993216, 4096 00020000:00200000:1.0:1789760766.091773:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.091793:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6345, 6345] 00000008:00010000:1.0:1789760766.091812:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.091819:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760766.091831:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25989120, 25993216) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.091845:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25989120=0x18c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.091905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.091908:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25993216 00000020:00010000:1.0:1789760766.091916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.091920:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.091922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.091925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.091928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.091938:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.091945:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.091959:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760766.092063:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 25989120<>25993216 00000100:00100000:1.0:1789760766.092108:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.092126:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796352:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760766.153138:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973664 gen 3 00000008:00010000:1.0:1789760766.153200:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:1.0:1789760766.153210:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.153221:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.153247:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:1.0:1789760766.153286:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796352:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760766.153427:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6345, 6345] 00000080:00200000:3.0:1789760766.153444:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760766.153450:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760766.153474:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25993216 i_blocks 51207 00000020:00200000:3.0:1789760766.153478:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760766.153489:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760766.153522:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25993216, count: 4096 00000080:00200000:3.0:1789760766.153528:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25993216, bytes: 4096 00000080:00200000:3.0:1789760766.153539:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760766.153548:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760766.153550:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25993216, 25997312) 0 0 00000080:00200000:3.0:1789760766.153554:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760766.153569:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6346, 6346] 00020000:00200000:3.0:1789760766.153578:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25993216 chunk: [25993216, 25997312) 25997312, 4096 00020000:00200000:3.0:1789760766.153583:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760766.153604:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6346, 6346] 00000008:00010000:3.0:1789760766.153624:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760766.153631:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760766.153642:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25993216, 25997312) append 0 append lockpos 0 00000080:00200000:3.0:1789760766.153655:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25993216=0x18ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760766.153712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760766.153716:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25997312 00000020:00010000:3.0:1789760766.153724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760766.153728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760766.153730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760766.153733:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760766.153736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760766.153744:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760766.153751:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760766.153767:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760766.153952:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 25993216<>25997312 00000100:00100000:3.0:1789760766.154003:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760766.154021:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493796480:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760766.188949:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760766.209676:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973665 gen 3 00000008:00010000:3.0:1789760766.209731:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760766.209740:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.209751:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.209881:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760766.209914:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493796480:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760766.210010:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6346, 6346] 00000080:00200000:1.0:1789760766.210027:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.210033:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.210048:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25997312 i_blocks 51207 00000020:00200000:1.0:1789760766.210052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.210062:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.210096:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25997312, count: 4096 00000080:00200000:1.0:1789760766.210104:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25997312, bytes: 4096 00000080:00200000:1.0:1789760766.210114:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.210124:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.210126:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25997312, 26001408) 0 0 00000080:00200000:1.0:1789760766.210131:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.210633:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6347, 6347] 00020000:00200000:1.0:1789760766.210646:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25997312 chunk: [25997312, 26001408) 26001408, 4096 00020000:00200000:1.0:1789760766.210650:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.210672:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6347, 6347] 00000008:00010000:1.0:1789760766.210691:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.210696:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760766.210707:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25997312, 26001408) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.210720:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=25997312=0x18cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.210967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.210971:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26001408 00000020:00010000:1.0:1789760766.210980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.210984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.210987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.210991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.210993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.211006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.211013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.211031:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760766.211230:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 25997312<>26001408 00000100:00100000:0.0:1789760766.211315:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796608:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760766.243608:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973666 gen 3 00000008:00010000:0.0:1789760766.243663:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760766.243671:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.243680:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.243702:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000080:00200000:2.0:1789760766.243807:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6347, 6347] 00000080:00200000:2.0:1789760766.243826:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.243833:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.243849:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26001408 i_blocks 51207 00000020:00200000:2.0:1789760766.243854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.243864:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.243896:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26001408, count: 4096 00000080:00200000:2.0:1789760766.243905:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26001408, bytes: 4096 00000080:00200000:2.0:1789760766.243915:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.243924:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.243926:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26001408, 26005504) 0 0 00000080:00200000:2.0:1789760766.243931:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.243948:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6348, 6348] 00020000:00200000:2.0:1789760766.243957:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26001408 chunk: [26001408, 26005504) 26005504, 4096 00020000:00200000:2.0:1789760766.243962:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.243981:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6348, 6348] 00000008:00010000:2.0:1789760766.243998:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.244004:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.244016:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26001408, 26005504) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.244029:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26001408=0x18cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.244078:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.244081:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26005504 00000020:00010000:2.0:1789760766.244089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.244093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.244096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.244099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.244102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.244110:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.244117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760766.244180:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796608:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760766.244237:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:0.0:1789760766.244338:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 26001408<>26005504 00000100:00100000:0.0:1789760766.244377:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.244391:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796736:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760766.291448:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:0.0:1789760766.291873:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.291887:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.291908:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:0.0:1789760766.291940:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493796736:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.292034:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6348, 6348] 00000080:00200000:2.0:1789760766.292051:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.292058:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.292073:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26005504 i_blocks 51207 00000020:00200000:2.0:1789760766.292076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.292085:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.292116:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26005504, count: 4096 00000080:00200000:2.0:1789760766.292123:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26005504, bytes: 4096 00000080:00200000:2.0:1789760766.292135:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.292154:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.292157:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26005504, 26009600) 0 0 00000080:00200000:2.0:1789760766.292162:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.292179:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6349, 6349] 00020000:00200000:2.0:1789760766.292187:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26005504 chunk: [26005504, 26009600) 26009600, 4096 00020000:00200000:2.0:1789760766.292192:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.292211:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6349, 6349] 00000008:00010000:2.0:1789760766.292237:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.292243:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.292260:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26005504, 26009600) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.292274:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26005504=0x18cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.292331:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.292335:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26009600 00000020:00010000:2.0:1789760766.292343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.292346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.292349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.292353:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.292356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.292364:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.292371:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.292388:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760766.293375:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26005504<>26009600 00000100:00100000:2.0:1789760766.293450:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760766.293485:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493796864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.342053:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973668 gen 3 00000008:00010000:3.0:1789760766.344492:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760766.344511:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.344528:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.344561:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760766.344601:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493796864:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.344965:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6349, 6349] 00000080:00200000:2.0:1789760766.344983:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.344989:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.345007:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26009600 i_blocks 51207 00000020:00200000:2.0:1789760766.345011:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.345025:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.345060:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26009600, count: 4096 00000080:00200000:2.0:1789760766.345068:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26009600, bytes: 4096 00000080:00200000:2.0:1789760766.345083:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.345091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.345094:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26009600, 26013696) 0 0 00000080:00200000:2.0:1789760766.345117:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.345135:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6350, 6350] 00020000:00200000:2.0:1789760766.345153:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26009600 chunk: [26009600, 26013696) 26013696, 4096 00020000:00200000:2.0:1789760766.345158:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.345189:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6350, 6350] 00000008:00010000:2.0:1789760766.345371:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.345378:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.345391:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26009600, 26013696) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.345446:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26009600=0x18ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.345521:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.345525:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26013696 00000020:00010000:2.0:1789760766.345533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.345537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.345540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.345544:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.345547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.345555:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.345562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.345578:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760766.346280:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26009600<>26013696 00000100:00100000:3.0:1789760766.346651:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493796992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.408255:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973669 gen 3 00000008:00010000:3.0:1789760766.408306:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760766.408315:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.408328:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.408350:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760766.408379:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493796992:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.408729:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6350, 6350] 00000080:00200000:2.0:1789760766.408754:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.408761:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.408777:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26013696 i_blocks 51207 00000020:00200000:2.0:1789760766.408780:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.408792:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.408828:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26013696, count: 4096 00000080:00200000:2.0:1789760766.408835:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26013696, bytes: 4096 00000080:00200000:2.0:1789760766.408844:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.408854:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.408856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26013696, 26017792) 0 0 00000080:00200000:2.0:1789760766.408861:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.408878:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6351, 6351] 00020000:00200000:2.0:1789760766.408886:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26013696 chunk: [26013696, 26017792) 26017792, 4096 00020000:00200000:2.0:1789760766.408891:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.408912:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6351, 6351] 00000008:00010000:2.0:1789760766.408929:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.408935:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.408948:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26013696, 26017792) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.408962:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26013696=0x18cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.409019:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.409022:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26017792 00000020:00010000:2.0:1789760766.409031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.409034:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.409038:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.409041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.409044:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.409051:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.409058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.409074:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760766.409260:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26013696<>26017792 00000100:00100000:3.0:1789760766.409431:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.456906:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973670 gen 3 00000008:00010000:3.0:1789760766.456962:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760766.456972:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.456984:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.457005:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760766.457037:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797120:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.457350:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6351, 6351] 00000080:00200000:2.0:1789760766.457366:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.457372:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.457389:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26017792 i_blocks 51207 00000020:00200000:2.0:1789760766.457423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.457435:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.457469:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26017792, count: 4096 00000080:00200000:2.0:1789760766.457476:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26017792, bytes: 4096 00000080:00200000:2.0:1789760766.457486:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.457494:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.457497:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26017792, 26021888) 0 0 00000080:00200000:2.0:1789760766.457502:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.457519:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6352, 6352] 00020000:00200000:2.0:1789760766.457527:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26017792 chunk: [26017792, 26021888) 26021888, 4096 00020000:00200000:2.0:1789760766.457531:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.457628:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6352, 6352] 00000008:00010000:2.0:1789760766.457647:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.457654:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.457667:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26017792, 26021888) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.457682:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26017792=0x18d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.457741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.457744:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26021888 00000020:00010000:2.0:1789760766.457753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.457756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.457760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.457763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.457766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.457782:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.457789:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.457805:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760766.457910:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26017792<>26021888 00000100:00100000:3.0:1789760766.458012:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760766.458033:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.498386:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973671 gen 3 00000008:00010000:3.0:1789760766.498475:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760766.498485:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.498495:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.498518:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760766.498557:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797248:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.498757:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6352, 6352] 00000080:00200000:2.0:1789760766.498775:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.498781:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.498796:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26021888 i_blocks 51207 00000020:00200000:2.0:1789760766.498799:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.498810:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.498908:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26021888, count: 4096 00000080:00200000:2.0:1789760766.498916:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26021888, bytes: 4096 00000080:00200000:2.0:1789760766.498926:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.498935:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.498938:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26021888, 26025984) 0 0 00000080:00200000:2.0:1789760766.498943:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.498960:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6353, 6353] 00020000:00200000:2.0:1789760766.498969:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26021888 chunk: [26021888, 26025984) 26025984, 4096 00020000:00200000:2.0:1789760766.498974:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.498994:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6353, 6353] 00000008:00010000:2.0:1789760766.499013:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.499019:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.499030:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26021888, 26025984) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.499045:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26021888=0x18d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.499105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.499108:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26025984 00000020:00010000:2.0:1789760766.499116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.499120:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.499122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.499127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.499129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.499137:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.499304:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.499322:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760766.499458:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26021888<>26025984 00000100:00100000:3.0:1789760766.500009:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760766.500034:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493797376:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.545267:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973672 gen 3 00000008:00010000:3.0:1789760766.545318:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760766.545327:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.545339:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.545361:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760766.545439:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493797376:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.545738:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6353, 6353] 00000080:00200000:2.0:1789760766.545756:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.545762:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.545777:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26025984 i_blocks 51207 00000020:00200000:2.0:1789760766.545781:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.545793:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.545839:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26025984, count: 4096 00000080:00200000:2.0:1789760766.545846:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26025984, bytes: 4096 00000080:00200000:2.0:1789760766.545856:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.545864:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.545866:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26025984, 26030080) 0 0 00000080:00200000:2.0:1789760766.545871:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.545887:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6354, 6354] 00020000:00200000:2.0:1789760766.545896:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26025984 chunk: [26025984, 26030080) 26030080, 4096 00020000:00200000:2.0:1789760766.545901:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.545922:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6354, 6354] 00000008:00010000:2.0:1789760766.545942:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.545947:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.545959:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26025984, 26030080) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.545973:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26025984=0x18d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.546033:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.546037:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26030080 00000020:00010000:2.0:1789760766.546047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.546050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.546054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.546057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.546061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.546070:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.546077:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.546100:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760766.546213:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26025984<>26030080 00000100:00100000:3.0:1789760766.546653:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760766.546676:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797504:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.601699:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973673 gen 3 00000008:00010000:3.0:1789760766.601755:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760766.601764:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.601787:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.601806:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760766.601839:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797504:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.602153:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6354, 6354] 00000080:00200000:2.0:1789760766.602170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.602177:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.602192:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26030080 i_blocks 51207 00000020:00200000:2.0:1789760766.602195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.602215:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.602256:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26030080, count: 4096 00000080:00200000:2.0:1789760766.602263:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26030080, bytes: 4096 00000080:00200000:2.0:1789760766.602274:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.602283:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.602285:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26030080, 26034176) 0 0 00000080:00200000:2.0:1789760766.602290:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.602305:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6355, 6355] 00020000:00200000:2.0:1789760766.602314:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26030080 chunk: [26030080, 26034176) 26034176, 4096 00020000:00200000:2.0:1789760766.602319:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.602532:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6355, 6355] 00000008:00010000:2.0:1789760766.602551:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.602556:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.602568:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26030080, 26034176) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.602581:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26030080=0x18d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.602640:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.602643:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26034176 00000020:00010000:2.0:1789760766.602651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.602654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.602657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.602661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.602664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.602673:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.602680:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.602696:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760766.602809:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26030080<>26034176 00000100:00100000:3.0:1789760766.603070:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797632:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.694447:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973674 gen 3 00000008:00010000:3.0:1789760766.694499:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760766.694509:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.694521:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.694547:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760766.694579:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797632:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760766.694871:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6355, 6355] 00000080:00200000:2.0:1789760766.694886:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.694890:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.694903:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26034176 i_blocks 51207 00000020:00200000:2.0:1789760766.694906:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.694916:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.694948:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26034176, count: 4096 00000080:00200000:2.0:1789760766.694954:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26034176, bytes: 4096 00000080:00200000:2.0:1789760766.694966:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.694973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.694975:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26034176, 26038272) 0 0 00000080:00200000:2.0:1789760766.694979:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.694995:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6356, 6356] 00020000:00200000:2.0:1789760766.695004:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26034176 chunk: [26034176, 26038272) 26038272, 4096 00020000:00200000:2.0:1789760766.695008:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.695028:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6356, 6356] 00000008:00010000:2.0:1789760766.695046:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.695052:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760766.695063:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26034176, 26038272) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.695077:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26034176=0x18d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.695134:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.695137:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26038272 00000020:00010000:2.0:1789760766.695159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.695163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.695167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.695171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.695174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.695182:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.695188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.695205:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760766.695678:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26034176<>26038272 00000100:00100000:2.0:1789760766.695716:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760766.695734:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797760:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.771228:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973675 gen 3 00000008:00010000:3.0:1789760766.771281:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760766.771290:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.771302:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.771331:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760766.771361:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493797760:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760766.771509:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6356, 6356] 00000080:00200000:0.0:1789760766.771526:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.771532:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.771548:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26038272 i_blocks 51207 00000020:00200000:0.0:1789760766.771551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.771562:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.771605:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26038272, count: 4096 00000080:00200000:0.0:1789760766.771613:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26038272, bytes: 4096 00000080:00200000:0.0:1789760766.771747:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.771756:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.771758:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26038272, 26042368) 0 0 00000080:00200000:0.0:1789760766.771763:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.771779:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6357, 6357] 00020000:00200000:0.0:1789760766.771787:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26038272 chunk: [26038272, 26042368) 26042368, 4096 00020000:00200000:0.0:1789760766.771792:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.771815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6357, 6357] 00000008:00010000:0.0:1789760766.771834:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.771841:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760766.771852:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26038272, 26042368) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.771868:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26038272=0x18d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.771931:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.771934:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26042368 00000020:00010000:0.0:1789760766.771943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.771947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.771950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.771954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.771957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.771966:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.771973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.771992:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760766.772103:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26038272<>26042368 00000100:00100000:0.0:1789760766.772163:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760766.772180:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493797888:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760766.834789:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973676 gen 3 00000008:00010000:0.0:1789760766.834831:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760766.834838:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.834846:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.834864:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760766.834889:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493797888:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760766.835053:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6357, 6357] 00000080:00200000:3.0:1789760766.835066:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760766.835071:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760766.835082:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26042368 i_blocks 51207 00000020:00200000:3.0:1789760766.835085:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760766.835093:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760766.835119:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26042368, count: 4096 00000080:00200000:3.0:1789760766.835124:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26042368, bytes: 4096 00000080:00200000:3.0:1789760766.835132:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760766.835138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760766.835150:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26042368, 26046464) 0 0 00000080:00200000:3.0:1789760766.835155:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760766.835167:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6358, 6358] 00020000:00200000:3.0:1789760766.835174:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26042368 chunk: [26042368, 26046464) 26046464, 4096 00020000:00200000:3.0:1789760766.835177:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760766.835200:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6358, 6358] 00000008:00010000:3.0:1789760766.835214:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760766.835219:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760766.835227:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26042368, 26046464) append 0 append lockpos 0 00000080:00200000:3.0:1789760766.835236:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26042368=0x18d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760766.835275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760766.835277:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26046464 00000020:00010000:3.0:1789760766.835283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760766.835286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760766.835289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760766.835291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760766.835294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760766.835300:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760766.835305:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760766.835319:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760766.835433:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 26042368<>26046464 00000100:00100000:3.0:1789760766.835476:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760766.835490:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493798016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760766.880734:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973677 gen 3 00000008:00010000:3.0:1789760766.880770:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760766.880776:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.880784:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.880806:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760766.880842:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493798016:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760766.880950:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6358, 6358] 00000080:00200000:0.0:1789760766.880963:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.880968:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.880980:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26046464 i_blocks 51207 00000020:00200000:0.0:1789760766.880984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.880992:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.881018:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26046464, count: 4096 00000080:00200000:0.0:1789760766.881023:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26046464, bytes: 4096 00000080:00200000:0.0:1789760766.881031:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.881038:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.881041:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26046464, 26050560) 0 0 00000080:00200000:0.0:1789760766.881044:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.881057:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6359, 6359] 00020000:00200000:0.0:1789760766.881064:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26046464 chunk: [26046464, 26050560) 26050560, 4096 00020000:00200000:0.0:1789760766.881067:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.881081:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6359, 6359] 00000008:00010000:0.0:1789760766.881093:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.881098:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760766.881107:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26046464, 26050560) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.881117:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26046464=0x18d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.881175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.881178:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26050560 00000020:00010000:0.0:1789760766.881184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.881187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.881190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.881203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.881207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.881213:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.881219:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.881233:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760766.881310:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26046464<>26050560 00000100:00100000:1.0:1789760766.881453:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760766.881484:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798144:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760766.956987:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973678 gen 3 00000008:00010000:1.0:1789760766.957036:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760766.957044:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760766.957054:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760766.957082:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760766.957110:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798144:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760766.957437:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6359, 6359] 00000080:00200000:0.0:1789760766.957453:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.957460:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.957477:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26050560 i_blocks 51207 00000020:00200000:0.0:1789760766.957482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.957493:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.957530:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26050560, count: 4096 00000080:00200000:0.0:1789760766.957538:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26050560, bytes: 4096 00000080:00200000:0.0:1789760766.957548:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.957556:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.957558:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26050560, 26054656) 0 0 00000080:00200000:0.0:1789760766.957563:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.957581:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6360, 6360] 00020000:00200000:0.0:1789760766.957589:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26050560 chunk: [26050560, 26054656) 26054656, 4096 00020000:00200000:0.0:1789760766.957594:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.957616:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6360, 6360] 00000008:00010000:0.0:1789760766.957636:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.957642:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760766.957655:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26050560, 26054656) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.957668:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26050560=0x18d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.957738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.957742:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26054656 00000020:00010000:0.0:1789760766.957750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.957754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.957757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.957760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.957763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.957781:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.957789:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.957806:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760766.961587:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 26050560<>26054656 00000100:00100000:1.0:1789760766.961694:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798272:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760767.012604:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973679 gen 3 00000008:00010000:1.0:1789760767.012671:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760767.012680:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.012692:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.012714:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760767.012743:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798272:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760767.012915:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6360, 6360] 00000080:00200000:3.0:1789760767.012933:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.012939:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.012955:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26054656 i_blocks 51207 00000020:00200000:3.0:1789760767.012959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.012969:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.013003:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26054656, count: 4096 00000080:00200000:3.0:1789760767.013011:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26054656, bytes: 4096 00000080:00200000:3.0:1789760767.013021:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.013030:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.013033:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26054656, 26058752) 0 0 00000080:00200000:3.0:1789760767.013037:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.013056:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6361, 6361] 00020000:00200000:3.0:1789760767.013064:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26054656 chunk: [26054656, 26058752) 26058752, 4096 00020000:00200000:3.0:1789760767.013069:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.013089:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6361, 6361] 00000008:00010000:3.0:1789760767.013108:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.013114:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760767.013127:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26054656, 26058752) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.013154:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26054656=0x18d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.013213:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.013216:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26058752 00000020:00010000:3.0:1789760767.013225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.013228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.013231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.013235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.013297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.013307:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.013314:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.013331:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760767.013593:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 26054656<>26058752 00000100:00100000:2.0:1789760767.013658:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760767.014374:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493798400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.076667:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973680 gen 3 00000008:00010000:3.0:1789760767.076721:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760767.076731:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.076742:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.076762:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760767.076804:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493798400:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.077005:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6361, 6361] 00000080:00200000:2.0:1789760767.077022:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.077028:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.077043:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26058752 i_blocks 51207 00000020:00200000:2.0:1789760767.077047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.077057:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.077092:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26058752, count: 4096 00000080:00200000:2.0:1789760767.077100:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26058752, bytes: 4096 00000080:00200000:2.0:1789760767.077110:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.077120:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.077122:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26058752, 26062848) 0 0 00000080:00200000:2.0:1789760767.077126:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.077154:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6362, 6362] 00020000:00200000:2.0:1789760767.077163:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26058752 chunk: [26058752, 26062848) 26062848, 4096 00020000:00200000:2.0:1789760767.077167:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.077189:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6362, 6362] 00000008:00010000:2.0:1789760767.077207:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.077214:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.077225:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26058752, 26062848) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.077239:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26058752=0x18da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.077295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.077298:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26062848 00000020:00010000:2.0:1789760767.077307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.077311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.077314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.077318:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.077321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.077329:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.077336:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.077352:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760767.077968:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26058752<>26062848 00000100:00100000:2.0:1789760767.078021:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760767.078044:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493798528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.150055:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973681 gen 3 00000008:00010000:3.0:1789760767.150100:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760767.150109:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.150191:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.150275:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760767.150307:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493798528:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760767.150322:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6362, 6362] 00000080:00200000:1.0:1789760767.150341:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760767.150347:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760767.150362:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26062848 i_blocks 51207 00000020:00200000:1.0:1789760767.150367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760767.150379:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760767.150441:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26062848, count: 4096 00000080:00200000:1.0:1789760767.150449:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26062848, bytes: 4096 00000080:00200000:1.0:1789760767.150459:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760767.150469:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760767.150471:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26062848, 26066944) 0 0 00000080:00200000:1.0:1789760767.150476:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760767.150492:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6363, 6363] 00020000:00200000:1.0:1789760767.150500:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26062848 chunk: [26062848, 26066944) 26066944, 4096 00020000:00200000:1.0:1789760767.150505:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760767.150525:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6363, 6363] 00000008:00010000:1.0:1789760767.151036:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760767.151044:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760767.151058:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26062848, 26066944) append 0 append lockpos 0 00000080:00200000:1.0:1789760767.151073:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26062848=0x18db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760767.151136:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760767.151151:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26066944 00000020:00010000:1.0:1789760767.151160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760767.151163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.151166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.151170:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760767.151173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760767.151185:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760767.151192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.151209:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760767.151315:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26062848<>26066944 00000100:00100000:1.0:1789760767.151365:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760767.151386:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798656:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760767.212648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760767.228631:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973682 gen 3 00000008:00010000:1.0:1789760767.228685:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760767.228694:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.228706:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.228733:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760767.228761:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493798656:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760767.228839:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6363, 6363] 00000080:00200000:3.0:1789760767.228856:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.228863:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.228878:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26066944 i_blocks 51207 00000020:00200000:3.0:1789760767.228882:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.228892:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.228926:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26066944, count: 4096 00000080:00200000:3.0:1789760767.228935:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26066944, bytes: 4096 00000080:00200000:3.0:1789760767.228950:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.228960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.228962:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26066944, 26071040) 0 0 00000080:00200000:3.0:1789760767.228967:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.228983:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6364, 6364] 00020000:00200000:3.0:1789760767.228992:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26066944 chunk: [26066944, 26071040) 26071040, 4096 00020000:00200000:3.0:1789760767.228996:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.229016:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6364, 6364] 00000008:00010000:3.0:1789760767.229033:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.229040:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760767.229052:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26066944, 26071040) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.229066:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26066944=0x18dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.229118:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.229120:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26071040 00000020:00010000:3.0:1789760767.229128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.229132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.229135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.229148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.229152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.229160:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.229167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.230264:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:1.0:1789760767.230527:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26066944<>26071040 00000100:00100000:0.0:1789760767.230594:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760767.231738:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493798784:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760767.278300:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973683 gen 3 00000008:00010000:0.0:1789760767.278358:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:0.0:1789760767.278368:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760767.278380:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760767.278433:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:0.0:1789760767.278462:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493798784:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760767.278570:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6364, 6364] 00000080:00200000:3.0:1789760767.278588:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.278594:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.278610:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26071040 i_blocks 51207 00000020:00200000:3.0:1789760767.278623:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.278633:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.278669:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26071040, count: 4096 00000080:00200000:3.0:1789760767.278677:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26071040, bytes: 4096 00000080:00200000:3.0:1789760767.278687:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.278695:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.278698:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26071040, 26075136) 0 0 00000080:00200000:3.0:1789760767.278702:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.278719:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6365, 6365] 00020000:00200000:3.0:1789760767.278728:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26071040 chunk: [26071040, 26075136) 26075136, 4096 00020000:00200000:3.0:1789760767.278732:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.278750:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6365, 6365] 00000008:00010000:3.0:1789760767.278766:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.278784:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760767.278796:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26071040, 26075136) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.278818:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26071040=0x18dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.278879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.278882:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26075136 00000020:00010000:3.0:1789760767.278890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.278893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.278907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.278912:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.278915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.278923:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.278930:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.278947:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760767.279053:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 26071040<>26075136 00000100:00100000:3.0:1789760767.279103:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760767.279121:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493798912:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760767.324676:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760767.324692:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.324704:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.324728:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760767.324761:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493798912:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760767.324809:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6365, 6365] 00000080:00200000:1.0:1789760767.324827:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760767.324833:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760767.324849:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26075136 i_blocks 51207 00000020:00200000:1.0:1789760767.324853:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760767.324863:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760767.324897:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26075136, count: 4096 00000080:00200000:1.0:1789760767.324906:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26075136, bytes: 4096 00000080:00200000:1.0:1789760767.324917:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760767.324926:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760767.324928:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26075136, 26079232) 0 0 00000080:00200000:1.0:1789760767.324933:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760767.324950:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6366, 6366] 00020000:00200000:1.0:1789760767.324958:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26075136 chunk: [26075136, 26079232) 26079232, 4096 00020000:00200000:1.0:1789760767.324963:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760767.324984:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6366, 6366] 00000008:00010000:1.0:1789760767.325004:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760767.325010:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760767.325021:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26075136, 26079232) append 0 append lockpos 0 00000080:00200000:1.0:1789760767.325035:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26075136=0x18de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760767.325094:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760767.325097:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26079232 00000020:00010000:1.0:1789760767.325105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760767.325109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.325112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.325115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760767.325118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760767.325130:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760767.325147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.325162:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760767.325268:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 26075136<>26079232 00000100:00100000:1.0:1789760767.325313:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760767.325329:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493799040:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760767.377691:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973684 gen 3 00000100:00100000:1.0:1789760767.377738:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016d880 x1876699493799040/t4294973685(4294973685) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760767.377790:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760767.377798:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.377808:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.377835:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760767.377857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493799040:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760767.377953:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6366, 6366] 00000080:00200000:3.0:1789760767.377968:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.377974:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.377987:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26079232 i_blocks 51207 00000020:00200000:3.0:1789760767.377991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.378000:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.378031:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26079232, count: 4096 00000080:00200000:3.0:1789760767.378038:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26079232, bytes: 4096 00000080:00200000:3.0:1789760767.378049:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.378058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.378060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26079232, 26083328) 0 0 00000080:00200000:3.0:1789760767.378065:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.378081:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6367, 6367] 00020000:00200000:3.0:1789760767.378089:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26079232 chunk: [26079232, 26083328) 26083328, 4096 00020000:00200000:3.0:1789760767.378093:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.378111:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6367, 6367] 00000008:00010000:3.0:1789760767.378130:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.378144:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760767.378157:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26079232, 26083328) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.378171:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26079232=0x18df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.378225:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.378228:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26083328 00000020:00010000:3.0:1789760767.378237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.378241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.378244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.378247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.378250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.378258:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.378264:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.378278:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760767.378367:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 26079232<>26083328 00000100:00100000:2.0:1789760767.378447:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.426117:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973686 gen 3 00000008:00010000:3.0:1789760767.426246:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760767.426255:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.426266:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.426290:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760767.426318:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799168:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760767.426429:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6367, 6367] 00000080:00200000:1.0:1789760767.426445:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760767.426452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760767.426467:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26083328 i_blocks 51207 00000020:00200000:1.0:1789760767.426471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760767.426480:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760767.426511:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26083328, count: 4096 00000080:00200000:1.0:1789760767.426517:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26083328, bytes: 4096 00000080:00200000:1.0:1789760767.426529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760767.426537:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760767.426540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26083328, 26087424) 0 0 00000080:00200000:1.0:1789760767.426544:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760767.426560:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6368, 6368] 00020000:00200000:1.0:1789760767.426567:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26083328 chunk: [26083328, 26087424) 26087424, 4096 00020000:00200000:1.0:1789760767.426572:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760767.426590:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6368, 6368] 00000008:00010000:1.0:1789760767.426607:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760767.426613:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760767.426688:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26083328, 26087424) append 0 append lockpos 0 00000080:00200000:1.0:1789760767.426760:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26083328=0x18e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760767.426883:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760767.426887:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26087424 00000020:00010000:1.0:1789760767.426895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760767.426899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.426902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.426907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760767.426909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760767.426920:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760767.426928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.426943:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760767.427041:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 26083328<>26087424 00000100:00100000:0.0:1789760767.427113:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493799296:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760767.459982:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973687 gen 3 00000008:00010000:0.0:1789760767.460031:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760767.460040:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760767.460050:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760767.460107:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760767.460176:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493799296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.460264:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6368, 6368] 00000080:00200000:2.0:1789760767.460281:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.460287:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.460303:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26087424 i_blocks 51207 00000020:00200000:2.0:1789760767.460307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.460317:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.460351:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26087424, count: 4096 00000080:00200000:2.0:1789760767.460358:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26087424, bytes: 4096 00000080:00200000:2.0:1789760767.460368:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.460377:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.460379:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26087424, 26091520) 0 0 00000080:00200000:2.0:1789760767.460384:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.460427:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6369, 6369] 00020000:00200000:2.0:1789760767.460436:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26087424 chunk: [26087424, 26091520) 26091520, 4096 00020000:00200000:2.0:1789760767.460441:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.460461:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6369, 6369] 00000008:00010000:2.0:1789760767.460479:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.460485:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.460497:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26087424, 26091520) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.460510:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26087424=0x18e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.460567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.460570:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26091520 00000020:00010000:2.0:1789760767.460578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.460581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.460584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.460590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.460592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.460601:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.460608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.460734:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760767.460831:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26087424<>26091520 00000100:00100000:3.0:1789760767.460906:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493799424:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.499336:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973688 gen 3 00000008:00010000:3.0:1789760767.499386:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760767.499424:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.499437:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.499558:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760767.499588:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493799424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.499939:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6369, 6369] 00000080:00200000:2.0:1789760767.499956:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.499962:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.499979:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26091520 i_blocks 51207 00000020:00200000:2.0:1789760767.499983:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.499995:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.500032:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26091520, count: 4096 00000080:00200000:2.0:1789760767.500040:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26091520, bytes: 4096 00000080:00200000:2.0:1789760767.500050:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.500059:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.500061:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26091520, 26095616) 0 0 00000080:00200000:2.0:1789760767.500065:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.500086:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6370, 6370] 00020000:00200000:2.0:1789760767.500096:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26091520 chunk: [26091520, 26095616) 26095616, 4096 00020000:00200000:2.0:1789760767.500102:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.500253:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6370, 6370] 00000008:00010000:2.0:1789760767.500276:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.500282:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.500296:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26091520, 26095616) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.500310:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26091520=0x18e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.500369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.500372:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26095616 00000020:00010000:2.0:1789760767.500381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.500385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.500388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.500431:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.500435:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.500444:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.500452:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.500474:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760767.500577:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26091520<>26095616 00000100:00100000:3.0:1789760767.500707:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799552:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.532722:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799808:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760767.534878:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493799680:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760767.534991:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493799936:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760767.538195:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799808:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760767.541682:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493799680:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760767.541711:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493799936:192.168.203.160@tcp:400:kworker.0 00000008:00010000:3.0:1789760767.541862:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760767.541870:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.541880:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.541901:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760767.541924:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493799552:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.542066:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6370, 6370] 00000080:00200000:2.0:1789760767.542091:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.542098:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.542114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26095616 i_blocks 51207 00000020:00200000:2.0:1789760767.542117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.542128:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.542173:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26095616, count: 4096 00000080:00200000:2.0:1789760767.542180:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26095616, bytes: 4096 00000080:00200000:2.0:1789760767.542197:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.542206:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.542208:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26095616, 26099712) 0 0 00000080:00200000:2.0:1789760767.542213:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.542229:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6371, 6371] 00020000:00200000:2.0:1789760767.542237:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26095616 chunk: [26095616, 26099712) 26099712, 4096 00020000:00200000:2.0:1789760767.542251:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.542270:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6371, 6371] 00000008:00010000:2.0:1789760767.542289:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.542295:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.542308:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26095616, 26099712) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.542321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26095616=0x18e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.542377:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.542380:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26099712 00000020:00010000:2.0:1789760767.542389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.542416:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.542420:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.542424:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.542427:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.542434:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.542440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.542455:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760767.542560:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26095616<>26099712 00000100:00100000:2.0:1789760767.542600:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760767.542616:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.593576:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973690 gen 3 00000008:00010000:3.0:1789760767.593650:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760767.593658:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.593670:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.593764:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6371, 6371] 00000008:00010000:3.0:1789760767.593791:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000080:00200000:2.0:1789760767.593792:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.593798:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.593813:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26099712 i_blocks 51207 00000020:00200000:2.0:1789760767.593817:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.593827:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.593861:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26099712, count: 4096 00000080:00200000:2.0:1789760767.593870:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26099712, bytes: 4096 00000080:00200000:2.0:1789760767.593884:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.593893:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.593895:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26099712, 26103808) 0 0 00000080:00200000:2.0:1789760767.593900:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.593917:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6372, 6372] 00020000:00200000:2.0:1789760767.593935:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26099712 chunk: [26099712, 26103808) 26103808, 4096 00020000:00200000:2.0:1789760767.593940:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.593963:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6372, 6372] 00000100:00100000:3.0:1789760767.593975:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800064:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760767.593980:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.593986:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.593997:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26099712, 26103808) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.594011:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26099712=0x18e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.594070:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.594074:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26103808 00000020:00010000:2.0:1789760767.594082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.594085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.594089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.594092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.594095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.594103:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.594111:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.594126:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760767.594237:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26099712<>26103808 00000100:00100000:3.0:1789760767.594552:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760767.594569:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.631819:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973691 gen 3 00000008:00010000:3.0:1789760767.631865:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760767.631873:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.631884:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.631908:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760767.631945:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.632354:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6372, 6372] 00000080:00200000:2.0:1789760767.632371:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.632376:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.632414:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26103808 i_blocks 51207 00000020:00200000:2.0:1789760767.632419:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.632430:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.632464:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26103808, count: 4096 00000080:00200000:2.0:1789760767.632471:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26103808, bytes: 4096 00000080:00200000:2.0:1789760767.632482:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.632490:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.632492:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26103808, 26107904) 0 0 00000080:00200000:2.0:1789760767.632497:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.632512:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6373, 6373] 00020000:00200000:2.0:1789760767.632521:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26103808 chunk: [26103808, 26107904) 26107904, 4096 00020000:00200000:2.0:1789760767.632525:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.632542:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6373, 6373] 00000008:00010000:2.0:1789760767.632558:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.632564:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.632575:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26103808, 26107904) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.632587:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26103808=0x18e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.632704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.632707:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26107904 00000020:00010000:2.0:1789760767.632716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.632719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.632722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.632725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.632728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.632735:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.632742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.632758:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760767.632863:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26103808<>26107904 00000100:00100000:3.0:1789760767.633195:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760767.633215:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.670087:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973692 gen 3 00000008:00010000:3.0:1789760767.670147:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760767.670155:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.670169:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.670191:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760767.670219:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800320:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.670340:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6373, 6373] 00000080:00200000:2.0:1789760767.670356:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.670362:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.670377:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26107904 i_blocks 51207 00000020:00200000:2.0:1789760767.670380:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.670390:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.670439:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26107904, count: 4096 00000080:00200000:2.0:1789760767.670447:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26107904, bytes: 4096 00000080:00200000:2.0:1789760767.670458:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.670469:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.670471:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26107904, 26112000) 0 0 00000080:00200000:2.0:1789760767.670475:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.670491:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6374, 6374] 00020000:00200000:2.0:1789760767.670499:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26107904 chunk: [26107904, 26112000) 26112000, 4096 00020000:00200000:2.0:1789760767.670503:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.670522:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6374, 6374] 00000008:00010000:2.0:1789760767.670538:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.670544:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.670555:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26107904, 26112000) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.670569:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26107904=0x18e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.670621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.670624:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26112000 00000020:00010000:2.0:1789760767.670632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.670635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.670638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.670641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.670644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.670651:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.670657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.670672:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760767.670872:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26107904<>26112000 00000100:00100000:3.0:1789760767.670967:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760767.670987:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.712033:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973693 gen 3 00000008:00010000:3.0:1789760767.712085:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760767.712094:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.712108:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.712174:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760767.712203:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800448:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.712286:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6374, 6374] 00000080:00200000:2.0:1789760767.712303:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.712308:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.712324:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26112000 i_blocks 51207 00000020:00200000:2.0:1789760767.712327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.712338:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.712372:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26112000, count: 4096 00000080:00200000:2.0:1789760767.712379:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26112000, bytes: 4096 00000080:00200000:2.0:1789760767.712389:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.712536:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.712538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26112000, 26116096) 0 0 00000080:00200000:2.0:1789760767.712543:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.712561:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6375, 6375] 00020000:00200000:2.0:1789760767.712570:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26112000 chunk: [26112000, 26116096) 26116096, 4096 00020000:00200000:2.0:1789760767.712575:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.712596:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6375, 6375] 00000008:00010000:2.0:1789760767.712737:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.712744:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.712758:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26112000, 26116096) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.712773:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26112000=0x18e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.712833:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.712836:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26116096 00000020:00010000:2.0:1789760767.712844:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.712848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.712851:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.712855:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.712858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.712866:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.712874:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.712889:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760767.712998:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26112000<>26116096 00000100:00100000:3.0:1789760767.713065:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.785456:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973694 gen 3 00000008:00010000:3.0:1789760767.785512:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760767.785521:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.785534:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.785555:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760767.785585:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800576:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.785692:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6375, 6375] 00000080:00200000:2.0:1789760767.785710:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.785716:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.785732:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26116096 i_blocks 51207 00000020:00200000:2.0:1789760767.785736:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.785748:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.785784:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26116096, count: 4096 00000080:00200000:2.0:1789760767.785792:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26116096, bytes: 4096 00000080:00200000:2.0:1789760767.785803:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.785814:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.785816:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26116096, 26120192) 0 0 00000080:00200000:2.0:1789760767.785821:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.785840:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6376, 6376] 00020000:00200000:2.0:1789760767.785850:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26116096 chunk: [26116096, 26120192) 26120192, 4096 00020000:00200000:2.0:1789760767.785855:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.785876:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6376, 6376] 00000008:00010000:2.0:1789760767.785896:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.785902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.785913:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26116096, 26120192) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.785927:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26116096=0x18e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.785986:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.785989:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26120192 00000020:00010000:2.0:1789760767.785998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.786001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.786004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.786007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.786010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.786018:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.786024:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.786041:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760767.786154:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26116096<>26120192 00000100:00100000:3.0:1789760767.788816:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800704:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760767.822446:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760767.822463:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.822476:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.822495:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760767.822524:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800704:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.822832:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6376, 6376] 00000080:00200000:2.0:1789760767.822849:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.822856:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.822871:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26120192 i_blocks 51207 00000020:00200000:2.0:1789760767.822875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.822885:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.822922:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26120192, count: 4096 00000080:00200000:2.0:1789760767.822929:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26120192, bytes: 4096 00000080:00200000:2.0:1789760767.822939:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.822947:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.822950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26120192, 26124288) 0 0 00000080:00200000:2.0:1789760767.822955:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.822973:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6377, 6377] 00020000:00200000:2.0:1789760767.822982:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26120192 chunk: [26120192, 26124288) 26124288, 4096 00020000:00200000:2.0:1789760767.822986:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.823006:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6377, 6377] 00000008:00010000:2.0:1789760767.823025:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.823031:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.823043:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26120192, 26124288) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.823058:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26120192=0x18e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.823117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.823120:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26124288 00000020:00010000:2.0:1789760767.823128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.823131:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.823146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.823150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.823153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.823161:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.823168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.823195:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760767.823298:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26120192<>26124288 00000100:00100000:3.0:1789760767.823365:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800832:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.868501:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973695 gen 3 00000100:00100000:3.0:1789760767.868717:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81f80 x1876699493800832/t4294973696(4294973696) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760767.868768:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760767.868776:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.868786:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.868810:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760767.868836:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493800832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.869094:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6377, 6377] 00000080:00200000:2.0:1789760767.869111:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.869117:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.869142:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26124288 i_blocks 51207 00000020:00200000:2.0:1789760767.869146:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.869157:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.869189:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26124288, count: 4096 00000080:00200000:2.0:1789760767.869197:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26124288, bytes: 4096 00000080:00200000:2.0:1789760767.869206:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.869214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.869216:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26124288, 26128384) 0 0 00000080:00200000:2.0:1789760767.869221:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.869239:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6378, 6378] 00020000:00200000:2.0:1789760767.869249:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26124288 chunk: [26124288, 26128384) 26128384, 4096 00020000:00200000:2.0:1789760767.869253:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.869274:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6378, 6378] 00000008:00010000:2.0:1789760767.869292:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.869299:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.869311:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26124288, 26128384) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.869325:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26124288=0x18ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.869920:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.869925:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26128384 00000020:00010000:2.0:1789760767.869935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.869939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.869943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.869946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.869949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.869958:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.869965:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.869985:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760767.870090:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26124288<>26128384 00000100:00100000:3.0:1789760767.870222:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800960:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.915507:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973696 gen 3 00000100:00100000:3.0:1789760767.915600:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80380 x1876699493800960/t4294973697(4294973697) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760767.915647:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760767.915653:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.915662:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.915716:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760767.915738:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493800960:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.915798:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6378, 6378] 00000080:00200000:2.0:1789760767.915814:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.915819:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.915833:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26128384 i_blocks 51207 00000020:00200000:2.0:1789760767.915836:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.915846:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.915878:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26128384, count: 4096 00000080:00200000:2.0:1789760767.915884:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26128384, bytes: 4096 00000080:00200000:2.0:1789760767.915944:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.915953:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.915955:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26128384, 26132480) 0 0 00000080:00200000:2.0:1789760767.915960:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.915975:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6379, 6379] 00020000:00200000:2.0:1789760767.915983:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26128384 chunk: [26128384, 26132480) 26132480, 4096 00020000:00200000:2.0:1789760767.915988:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.916006:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6379, 6379] 00000008:00010000:2.0:1789760767.916023:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.916029:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.916040:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26128384, 26132480) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.916053:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26128384=0x18eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.916172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.916175:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26132480 00000020:00010000:2.0:1789760767.916183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.916187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.916190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.916193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.916196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.916203:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.916209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.916222:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760767.916372:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26128384<>26132480 00000100:00100000:3.0:1789760767.916429:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493801088:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.948154:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973697 gen 3 00000100:00100000:3.0:1789760767.948196:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80000 x1876699493801088/t4294973698(4294973698) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760767.948246:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760767.948254:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.948268:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.948329:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760767.948355:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493801088:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.948550:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6379, 6379] 00000080:00200000:2.0:1789760767.948568:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.948574:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.948594:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26132480 i_blocks 51207 00000020:00200000:2.0:1789760767.948598:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.948611:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.948648:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26132480, count: 4096 00000080:00200000:2.0:1789760767.948656:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26132480, bytes: 4096 00000080:00200000:2.0:1789760767.948666:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.948676:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.948678:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26132480, 26136576) 0 0 00000080:00200000:2.0:1789760767.948683:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.948702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6380, 6380] 00020000:00200000:2.0:1789760767.948711:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26132480 chunk: [26132480, 26136576) 26136576, 4096 00020000:00200000:2.0:1789760767.948716:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.948735:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6380, 6380] 00000008:00010000:2.0:1789760767.948754:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.948760:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.948773:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26132480, 26136576) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.948786:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26132480=0x18ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.948910:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.948913:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26136576 00000020:00010000:2.0:1789760767.948922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.948926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.948929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.948932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.948936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.948944:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.948952:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.948967:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760767.949077:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26132480<>26136576 00000100:00100000:3.0:1789760767.949158:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760767.978880:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973698 gen 3 00000100:00100000:3.0:1789760767.978923:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80a80 x1876699493801216/t4294973699(4294973699) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760783 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760767.978976:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760767.978984:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.978995:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.979015:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760767.979040:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801216:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760767.979142:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6380, 6380] 00000080:00200000:2.0:1789760767.979159:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.979164:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.979178:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26136576 i_blocks 51207 00000020:00200000:2.0:1789760767.979189:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.979200:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.979236:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26136576, count: 4096 00000080:00200000:2.0:1789760767.979242:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26136576, bytes: 4096 00000080:00200000:2.0:1789760767.979253:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.979262:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.979264:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26136576, 26140672) 0 0 00000080:00200000:2.0:1789760767.979270:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.979287:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6381, 6381] 00020000:00200000:2.0:1789760767.979295:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26136576 chunk: [26136576, 26140672) 26140672, 4096 00020000:00200000:2.0:1789760767.979300:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.979318:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6381, 6381] 00000008:00010000:2.0:1789760767.979336:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.979342:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760767.979354:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26136576, 26140672) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.979367:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26136576=0x18ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.979447:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.979449:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26140672 00000020:00010000:2.0:1789760767.979459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.979463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.979466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.979470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.979473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.979481:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.979487:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.979503:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760767.979609:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26136576<>26140672 00000100:00100000:3.0:1789760767.979863:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760767.979881:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.009252:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973700 gen 3 00000008:00010000:3.0:1789760768.009309:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760768.009318:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.009329:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.009347:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760768.009374:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.009497:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6381, 6381] 00000080:00200000:2.0:1789760768.009515:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.009520:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.009536:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26140672 i_blocks 51207 00000020:00200000:2.0:1789760768.009539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.009550:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.009583:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26140672, count: 4096 00000080:00200000:2.0:1789760768.009591:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26140672, bytes: 4096 00000080:00200000:2.0:1789760768.009600:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.009608:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.009611:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26140672, 26144768) 0 0 00000080:00200000:2.0:1789760768.009627:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.009643:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6382, 6382] 00020000:00200000:2.0:1789760768.009652:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26140672 chunk: [26140672, 26144768) 26144768, 4096 00020000:00200000:2.0:1789760768.009656:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.009677:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6382, 6382] 00000008:00010000:2.0:1789760768.009695:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.009700:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.009712:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26140672, 26144768) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.009729:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26140672=0x18ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.009784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.009787:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26144768 00000020:00010000:2.0:1789760768.009795:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.009798:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.009801:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.009804:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.009807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.009814:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.009832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.009847:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760768.009945:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26140672<>26144768 00000100:00100000:3.0:1789760768.010194:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.042885:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973701 gen 3 00000008:00010000:3.0:1789760768.042938:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760768.042947:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.042960:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.043021:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760768.043054:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801472:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.043336:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6382, 6382] 00000080:00200000:2.0:1789760768.043354:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.043360:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.043375:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26144768 i_blocks 51207 00000020:00200000:2.0:1789760768.043379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.043390:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.043458:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26144768, count: 4096 00000080:00200000:2.0:1789760768.043465:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26144768, bytes: 4096 00000080:00200000:2.0:1789760768.043476:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.043485:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.043487:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26144768, 26148864) 0 0 00000080:00200000:2.0:1789760768.043491:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.043509:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6383, 6383] 00020000:00200000:2.0:1789760768.043518:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26144768 chunk: [26144768, 26148864) 26148864, 4096 00020000:00200000:2.0:1789760768.043522:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.043542:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6383, 6383] 00000008:00010000:2.0:1789760768.043559:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.043565:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.043577:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26144768, 26148864) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.043591:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26144768=0x18ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.043720:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.043723:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26148864 00000020:00010000:2.0:1789760768.043732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.043736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.043738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.043741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.043744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.043751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.043759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.043775:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760768.043937:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26144768<>26148864 00000100:00100000:3.0:1789760768.044005:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493801600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.077066:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973702 gen 3 00000008:00010000:3.0:1789760768.077118:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760768.077128:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.077150:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.077170:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760768.077210:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493801600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.077358:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6383, 6383] 00000080:00200000:2.0:1789760768.077375:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.077383:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.077433:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26148864 i_blocks 51207 00000020:00200000:2.0:1789760768.077438:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.077451:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.077486:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26148864, count: 4096 00000080:00200000:2.0:1789760768.077494:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26148864, bytes: 4096 00000080:00200000:2.0:1789760768.077506:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.077516:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.077518:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26148864, 26152960) 0 0 00000080:00200000:2.0:1789760768.077524:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.077543:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6384, 6384] 00020000:00200000:2.0:1789760768.077551:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26148864 chunk: [26148864, 26152960) 26152960, 4096 00020000:00200000:2.0:1789760768.077556:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.077576:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6384, 6384] 00000008:00010000:2.0:1789760768.077595:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.077602:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.077615:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26148864, 26152960) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.077702:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26148864=0x18f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.077764:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.077767:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26152960 00000020:00010000:2.0:1789760768.077776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.077780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.077783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.077786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.077789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.077797:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.077805:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.077821:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760768.077926:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26148864<>26152960 00000100:00100000:3.0:1789760768.078035:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801728:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760768.110689:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760768.110704:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.110715:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.110734:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760768.110761:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801728:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.110938:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6384, 6384] 00000080:00200000:2.0:1789760768.110954:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.110960:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.110975:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26152960 i_blocks 51207 00000020:00200000:2.0:1789760768.110979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.110990:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.111024:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26152960, count: 4096 00000080:00200000:2.0:1789760768.111032:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26152960, bytes: 4096 00000080:00200000:2.0:1789760768.111041:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.111049:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.111052:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26152960, 26157056) 0 0 00000080:00200000:2.0:1789760768.111056:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.111073:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6385, 6385] 00020000:00200000:2.0:1789760768.111082:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26152960 chunk: [26152960, 26157056) 26157056, 4096 00020000:00200000:2.0:1789760768.111086:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.111105:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6385, 6385] 00000008:00010000:2.0:1789760768.111124:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.111130:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.111152:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26152960, 26157056) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.111166:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26152960=0x18f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.111231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.111234:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26157056 00000020:00010000:2.0:1789760768.111242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.111246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.111249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.111252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.111255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.111263:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.111270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.111284:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760768.111390:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26152960<>26157056 00000100:00100000:3.0:1789760768.111592:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760768.111672:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.171845:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973703 gen 3 00000100:00100000:3.0:1789760768.171896:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80380 x1876699493801856/t4294973704(4294973704) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760784 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760768.171945:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760768.171953:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.171964:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.171983:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760768.172006:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.172307:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6385, 6385] 00000080:00200000:2.0:1789760768.172324:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.172330:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.172346:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26157056 i_blocks 51207 00000020:00200000:2.0:1789760768.172350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.172360:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.172452:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26157056, count: 4096 00000080:00200000:2.0:1789760768.172460:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26157056, bytes: 4096 00000080:00200000:2.0:1789760768.172470:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.172478:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.172481:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26157056, 26161152) 0 0 00000080:00200000:2.0:1789760768.172485:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.172502:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6386, 6386] 00020000:00200000:2.0:1789760768.172512:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26157056 chunk: [26157056, 26161152) 26161152, 4096 00020000:00200000:2.0:1789760768.172517:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.172535:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6386, 6386] 00000008:00010000:2.0:1789760768.172552:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.172558:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.172570:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26157056, 26161152) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.172584:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26157056=0x18f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.172653:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.172656:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26161152 00000020:00010000:2.0:1789760768.172665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.172669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.172672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.172675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.172678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.172686:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.172693:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.172709:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760768.172811:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26157056<>26161152 00000100:00100000:3.0:1789760768.172881:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.215142:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973704 gen 3 00000100:00100000:3.0:1789760768.215203:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81f80 x1876699493801984/t4294973705(4294973705) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760784 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760768.215254:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760768.215262:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.215274:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.215295:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760768.215322:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493801984:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.215486:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6386, 6386] 00000080:00200000:0.0:1789760768.215504:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.215511:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.215526:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26161152 i_blocks 51207 00000020:00200000:0.0:1789760768.215531:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.215542:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.215582:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26161152, count: 4096 00000080:00200000:0.0:1789760768.215589:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26161152, bytes: 4096 00000080:00200000:0.0:1789760768.215599:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.215608:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.215622:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26161152, 26165248) 0 0 00000080:00200000:0.0:1789760768.215628:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.215647:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6387, 6387] 00020000:00200000:0.0:1789760768.215656:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26161152 chunk: [26161152, 26165248) 26165248, 4096 00020000:00200000:0.0:1789760768.215661:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.215682:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6387, 6387] 00000008:00010000:0.0:1789760768.215704:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.215710:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.215724:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26161152, 26165248) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.215740:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26161152=0x18f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.215801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.215805:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26165248 00000020:00010000:0.0:1789760768.215813:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.215816:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.215819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.215822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.215825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.215832:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.215839:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.215856:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760768.215973:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26161152<>26165248 00000100:00100000:0.0:1789760768.216021:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760768.216040:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802112:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760768.236816:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760768.253530:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973706 gen 3 00000008:00010000:0.0:1789760768.253598:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760768.253608:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.253628:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.253653:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760768.253681:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802112:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760768.253788:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6387, 6387] 00000080:00200000:1.0:1789760768.253805:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.253811:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.253838:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26165248 i_blocks 51207 00000020:00200000:1.0:1789760768.253842:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.253853:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.253885:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26165248, count: 4096 00000080:00200000:1.0:1789760768.253892:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26165248, bytes: 4096 00000080:00200000:1.0:1789760768.253902:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.253912:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.253914:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26165248, 26169344) 0 0 00000080:00200000:1.0:1789760768.253919:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.253935:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6388, 6388] 00020000:00200000:1.0:1789760768.253942:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26165248 chunk: [26165248, 26169344) 26169344, 4096 00020000:00200000:1.0:1789760768.253946:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.253965:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6388, 6388] 00000008:00010000:1.0:1789760768.253983:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.253990:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760768.254003:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26165248, 26169344) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.254017:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26165248=0x18f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.254076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.254079:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26169344 00000020:00010000:1.0:1789760768.254087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.254091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.254094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.254098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.254101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.254112:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.254119:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.254143:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760768.254256:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 26165248<>26169344 00000100:00100000:1.0:1789760768.254296:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760768.254312:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493802240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760768.323460:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973707 gen 3 00000008:00010000:1.0:1789760768.323514:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760768.323522:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.323535:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.323553:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760768.323580:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493802240:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760768.323674:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6388, 6388] 00000080:00200000:3.0:1789760768.323688:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760768.323693:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760768.323707:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26169344 i_blocks 51207 00000020:00200000:3.0:1789760768.323710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760768.323718:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760768.323748:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26169344, count: 4096 00000080:00200000:3.0:1789760768.323754:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26169344, bytes: 4096 00000080:00200000:3.0:1789760768.323763:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760768.323770:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760768.323771:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26169344, 26173440) 0 0 00000080:00200000:3.0:1789760768.323775:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760768.323787:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6389, 6389] 00020000:00200000:3.0:1789760768.323793:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26169344 chunk: [26169344, 26173440) 26173440, 4096 00020000:00200000:3.0:1789760768.323797:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760768.323813:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6389, 6389] 00000008:00010000:3.0:1789760768.323830:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760768.323837:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760768.323849:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26169344, 26173440) append 0 append lockpos 0 00000080:00200000:3.0:1789760768.323863:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26169344=0x18f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760768.323923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760768.323926:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26173440 00000020:00010000:3.0:1789760768.323935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760768.323939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.323942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.323945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760768.323948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760768.323956:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760768.323963:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760768.323980:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760768.324089:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 26169344<>26173440 00000100:00100000:2.0:1789760768.324170:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493802368:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760768.379487:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760768.379503:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.379514:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.379539:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760768.379627:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493802368:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760768.379663:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6389, 6389] 00000080:00200000:1.0:1789760768.379680:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.379687:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.379701:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26173440 i_blocks 51207 00000020:00200000:1.0:1789760768.379705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.379715:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.379746:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26173440, count: 4096 00000080:00200000:1.0:1789760768.379753:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26173440, bytes: 4096 00000080:00200000:1.0:1789760768.379763:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.379771:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.379773:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26173440, 26177536) 0 0 00000080:00200000:1.0:1789760768.379778:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.379793:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6390, 6390] 00020000:00200000:1.0:1789760768.379801:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26173440 chunk: [26173440, 26177536) 26177536, 4096 00020000:00200000:1.0:1789760768.379806:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.379824:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6390, 6390] 00000008:00010000:1.0:1789760768.379841:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.379846:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760768.379857:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26173440, 26177536) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.379871:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26173440=0x18f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.379927:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.379929:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26177536 00000020:00010000:1.0:1789760768.379937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.379941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.379944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.379948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.379951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.379963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.379969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.379984:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760768.380087:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 26173440<>26177536 00000100:00100000:0.0:1789760768.380222:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493802496:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760768.418863:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973709 gen 3 00000008:00010000:0.0:1789760768.418935:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760768.418944:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.418955:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.418982:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760768.419014:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493802496:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760768.419345:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6390, 6390] 00000080:00200000:1.0:1789760768.419363:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.419371:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.419388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26177536 i_blocks 51207 00000020:00200000:1.0:1789760768.419440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.419453:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.419487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26177536, count: 4096 00000080:00200000:1.0:1789760768.419495:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26177536, bytes: 4096 00000080:00200000:1.0:1789760768.419506:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.419515:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.419517:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26177536, 26181632) 0 0 00000080:00200000:1.0:1789760768.419522:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.419539:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6391, 6391] 00020000:00200000:1.0:1789760768.419548:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26177536 chunk: [26177536, 26181632) 26181632, 4096 00020000:00200000:1.0:1789760768.419553:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.419575:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6391, 6391] 00000008:00010000:1.0:1789760768.419593:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.419599:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760768.419610:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26177536, 26181632) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.419745:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26177536=0x18f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.419810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.419813:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26181632 00000020:00010000:1.0:1789760768.419824:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.419828:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.419831:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.419835:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.419838:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.419849:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.419857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.419872:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760768.419982:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 26177536<>26181632 00000100:00100000:0.0:1789760768.420076:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802624:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760768.456850:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973710 gen 3 00000008:00010000:0.0:1789760768.456904:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760768.456913:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.456923:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.456944:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760768.456976:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802624:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.457074:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6391, 6391] 00000080:00200000:2.0:1789760768.457091:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.457097:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.457113:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26181632 i_blocks 51207 00000020:00200000:2.0:1789760768.457117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.457126:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.457167:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26181632, count: 4096 00000080:00200000:2.0:1789760768.457175:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26181632, bytes: 4096 00000080:00200000:2.0:1789760768.457194:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.457202:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.457205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26181632, 26185728) 0 0 00000080:00200000:2.0:1789760768.457210:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.457224:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6392, 6392] 00020000:00200000:2.0:1789760768.457241:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26181632 chunk: [26181632, 26185728) 26185728, 4096 00020000:00200000:2.0:1789760768.457246:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.457267:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6392, 6392] 00000008:00010000:2.0:1789760768.457285:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.457291:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.457303:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26181632, 26185728) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.457316:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26181632=0x18f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.457375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.457378:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26185728 00000020:00010000:2.0:1789760768.457385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.457389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.457429:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.457434:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.457437:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.457446:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.457454:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.457469:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760768.457571:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26181632<>26185728 00000100:00100000:3.0:1789760768.457775:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493802752:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.530783:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973711 gen 3 00000008:00010000:3.0:1789760768.530842:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760768.530850:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.530860:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.530885:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760768.530914:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493802752:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.531207:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6392, 6392] 00000080:00200000:0.0:1789760768.531224:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.531229:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.531245:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26185728 i_blocks 51207 00000020:00200000:0.0:1789760768.531250:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.531262:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.531296:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26185728, count: 4096 00000080:00200000:0.0:1789760768.531303:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26185728, bytes: 4096 00000080:00200000:0.0:1789760768.531313:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.531322:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.531324:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26185728, 26189824) 0 0 00000080:00200000:0.0:1789760768.531329:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.531346:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6393, 6393] 00020000:00200000:0.0:1789760768.531355:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26185728 chunk: [26185728, 26189824) 26189824, 4096 00020000:00200000:0.0:1789760768.531360:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.531381:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6393, 6393] 00000008:00010000:0.0:1789760768.531431:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.531438:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.531449:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26185728, 26189824) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.531464:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26185728=0x18f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.531522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.531525:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26189824 00000020:00010000:0.0:1789760768.531533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.531536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.531539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.531542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.531545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.531552:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.531558:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.531574:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760768.531688:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26185728<>26189824 00000100:00100000:0.0:1789760768.531734:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760768.531751:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802880:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760768.564861:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973712 gen 3 00000008:00010000:0.0:1789760768.564913:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760768.564921:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.564930:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.564954:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760768.564983:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493802880:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760768.565087:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6393, 6393] 00000080:00200000:3.0:1789760768.565104:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760768.565111:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760768.565126:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26189824 i_blocks 51207 00000020:00200000:3.0:1789760768.565140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760768.565151:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760768.565195:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26189824, count: 4096 00000080:00200000:3.0:1789760768.565203:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26189824, bytes: 4096 00000080:00200000:3.0:1789760768.565214:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760768.565223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760768.565225:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26189824, 26193920) 0 0 00000080:00200000:3.0:1789760768.565229:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760768.565246:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6394, 6394] 00020000:00200000:3.0:1789760768.565255:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26189824 chunk: [26189824, 26193920) 26193920, 4096 00020000:00200000:3.0:1789760768.565259:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760768.565277:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6394, 6394] 00000008:00010000:3.0:1789760768.565294:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760768.565300:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760768.565324:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26189824, 26193920) append 0 append lockpos 0 00000080:00200000:3.0:1789760768.565337:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26189824=0x18fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760768.565391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760768.565438:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26193920 00000020:00010000:3.0:1789760768.565446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760768.565450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.565453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.565456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760768.565459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760768.565465:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760768.565472:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760768.565488:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760768.565592:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 26189824<>26193920 00000100:00100000:3.0:1789760768.565975:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760768.565995:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803008:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760768.610530:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760768.610546:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.610557:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.610579:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760768.610607:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803008:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.611222:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6394, 6394] 00000080:00200000:2.0:1789760768.611241:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.611248:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.611264:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26193920 i_blocks 51207 00000020:00200000:2.0:1789760768.611269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.611281:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.611313:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26193920, count: 4096 00000080:00200000:2.0:1789760768.611321:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26193920, bytes: 4096 00000080:00200000:2.0:1789760768.611333:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.611342:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.611346:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26193920, 26198016) 0 0 00000080:00200000:2.0:1789760768.611350:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.611368:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6395, 6395] 00020000:00200000:2.0:1789760768.611377:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26193920 chunk: [26193920, 26198016) 26198016, 4096 00020000:00200000:2.0:1789760768.611382:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.611433:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6395, 6395] 00000008:00010000:2.0:1789760768.611452:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.611459:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.611472:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26193920, 26198016) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.611486:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26193920=0x18fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.611546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.611549:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26198016 00000020:00010000:2.0:1789760768.611557:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.611561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.611564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.611567:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.611571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.611578:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.611585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.611602:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760768.611851:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26193920<>26198016 00000100:00100000:2.0:1789760768.611896:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760768.611918:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493803136:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.645894:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973713 gen 3 00000100:00100000:3.0:1789760768.645938:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83800 x1876699493803136/t4294973714(4294973714) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760784 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760768.645987:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760768.645994:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.646005:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.646024:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760768.646048:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493803136:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.646227:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6395, 6395] 00000080:00200000:2.0:1789760768.646243:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.646249:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.646264:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26198016 i_blocks 51207 00000020:00200000:2.0:1789760768.646268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.646279:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.646312:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26198016, count: 4096 00000080:00200000:2.0:1789760768.646320:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26198016, bytes: 4096 00000080:00200000:2.0:1789760768.646331:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.646340:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.646342:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26198016, 26202112) 0 0 00000080:00200000:2.0:1789760768.646347:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.646363:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6396, 6396] 00020000:00200000:2.0:1789760768.646372:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26198016 chunk: [26198016, 26202112) 26202112, 4096 00020000:00200000:2.0:1789760768.646377:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.646430:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6396, 6396] 00000008:00010000:2.0:1789760768.646449:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.646456:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.646468:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26198016, 26202112) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.646481:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26198016=0x18fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.646540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.646543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26202112 00000020:00010000:2.0:1789760768.646551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.646555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.646558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.646562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.646565:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.646573:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.646580:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.646597:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760768.646774:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26198016<>26202112 00000100:00100000:2.0:1789760768.646887:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760768.646908:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803264:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.681909:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973715 gen 3 00000008:00010000:3.0:1789760768.681976:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760768.681983:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.681994:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.682058:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760768.682090:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803264:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760768.682175:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6396, 6396] 00000080:00200000:2.0:1789760768.682758:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.682764:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.682780:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26202112 i_blocks 51207 00000020:00200000:2.0:1789760768.682785:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.682797:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.682832:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26202112, count: 4096 00000080:00200000:2.0:1789760768.682839:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26202112, bytes: 4096 00000080:00200000:2.0:1789760768.682851:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.682859:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.682862:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26202112, 26206208) 0 0 00000080:00200000:2.0:1789760768.682866:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.682882:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6397, 6397] 00020000:00200000:2.0:1789760768.682890:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26202112 chunk: [26202112, 26206208) 26206208, 4096 00020000:00200000:2.0:1789760768.682894:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.682912:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6397, 6397] 00000008:00010000:2.0:1789760768.682930:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.682937:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760768.682948:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26202112, 26206208) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.682963:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26202112=0x18fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.683019:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.683022:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26206208 00000020:00010000:2.0:1789760768.683030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.683033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.683036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.683040:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.683043:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.683050:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.683057:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.683076:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760768.683201:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26202112<>26206208 00000100:00100000:3.0:1789760768.683280:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803392:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760768.709928:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973716 gen 3 00000008:00010000:3.0:1789760768.709975:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760768.709984:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.709996:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.710019:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760768.710051:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493803392:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.710170:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6397, 6397] 00000080:00200000:0.0:1789760768.710187:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.710193:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.710208:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26206208 i_blocks 51207 00000020:00200000:0.0:1789760768.710211:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.710231:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.710264:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26206208, count: 4096 00000080:00200000:0.0:1789760768.710270:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26206208, bytes: 4096 00000080:00200000:0.0:1789760768.710279:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.710288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.710290:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26206208, 26210304) 0 0 00000080:00200000:0.0:1789760768.710295:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.710310:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6398, 6398] 00020000:00200000:0.0:1789760768.710318:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26206208 chunk: [26206208, 26210304) 26210304, 4096 00020000:00200000:0.0:1789760768.710322:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.710339:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6398, 6398] 00000008:00010000:0.0:1789760768.710355:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.710360:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.710371:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26206208, 26210304) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.710384:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26206208=0x18fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.710463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.710466:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26210304 00000020:00010000:0.0:1789760768.710474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.710477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.710480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.710485:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.710488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.710494:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.710500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.710515:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760768.710624:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26206208<>26210304 00000100:00100000:1.0:1789760768.710778:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760768.710796:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803520:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760768.752805:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973717 gen 3 00000008:00010000:1.0:1789760768.752873:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760768.752883:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.752895:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.752916:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760768.752947:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803520:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.753169:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6398, 6398] 00000080:00200000:0.0:1789760768.753196:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.753203:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.753311:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26210304 i_blocks 51207 00000020:00200000:0.0:1789760768.753316:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.753326:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.753360:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26210304, count: 4096 00000080:00200000:0.0:1789760768.753367:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26210304, bytes: 4096 00000080:00200000:0.0:1789760768.753377:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.753386:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.753388:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26210304, 26214400) 0 0 00000080:00200000:0.0:1789760768.753423:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.753439:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6399, 6399] 00020000:00200000:0.0:1789760768.753448:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26210304 chunk: [26210304, 26214400) 26214400, 4096 00020000:00200000:0.0:1789760768.753452:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.753472:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6399, 6399] 00000008:00010000:0.0:1789760768.753490:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.753496:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.753507:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26210304, 26214400) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.753519:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26210304=0x18ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.753568:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.753570:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26214400 00000020:00010000:0.0:1789760768.753578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.753581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.753584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.753588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.753591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.753598:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.753603:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.753627:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760768.753731:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 26210304<>26214400 00000100:00100000:1.0:1789760768.754048:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803648:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760768.792895:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973718 gen 3 00000008:00010000:1.0:1789760768.792966:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760768.792975:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.792985:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.793004:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760768.793033:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803648:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.793526:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6399, 6399] 00000080:00200000:0.0:1789760768.793543:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.793549:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.793567:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26214400 i_blocks 51207 00000020:00200000:0.0:1789760768.793570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.793582:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.793617:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26214400, count: 4096 00000080:00200000:0.0:1789760768.793624:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26214400, bytes: 4096 00000080:00200000:0.0:1789760768.793635:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.793644:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.793647:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26214400, 26218496) 0 0 00000080:00200000:0.0:1789760768.793653:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.793671:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6400, 6400] 00020000:00200000:0.0:1789760768.793680:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26214400 chunk: [26214400, 26218496) 26218496, 4096 00020000:00200000:0.0:1789760768.793685:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.793705:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6400, 6400] 00000008:00010000:0.0:1789760768.793725:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.793731:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.793744:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26214400, 26218496) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.793758:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26214400=0x1900000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.793834:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.793837:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26218496 00000020:00010000:0.0:1789760768.793844:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.793848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.793850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.793854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.793857:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.793864:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.793870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.793884:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760768.793983:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26214400<>26218496 00000100:00100000:1.0:1789760768.794565:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493803776:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760768.854156:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973719 gen 3 00000008:00010000:1.0:1789760768.854212:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760768.854600:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.854613:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.854692:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760768.854724:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493803776:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.855213:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6400, 6400] 00000080:00200000:0.0:1789760768.856168:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.856176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.856196:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26218496 i_blocks 51207 00000020:00200000:0.0:1789760768.856202:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.856216:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.856263:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26218496, count: 4096 00000080:00200000:0.0:1789760768.856279:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26218496, bytes: 4096 00000080:00200000:0.0:1789760768.856296:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.856305:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.856308:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26218496, 26222592) 0 0 00000080:00200000:0.0:1789760768.856313:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.856327:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6401, 6401] 00020000:00200000:0.0:1789760768.856336:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26218496 chunk: [26218496, 26222592) 26222592, 4096 00020000:00200000:0.0:1789760768.856340:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.856362:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6401, 6401] 00000008:00010000:0.0:1789760768.856382:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.856389:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.856418:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26218496, 26222592) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.856435:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26218496=0x1901000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.856500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.856657:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26222592 00000020:00010000:0.0:1789760768.856668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.856672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.856675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.856679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.856682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.856690:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.856697:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.856722:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760768.856837:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 26218496<>26222592 00000100:00100000:1.0:1789760768.856995:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760768.921805:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973720 gen 3 00000008:00010000:1.0:1789760768.922029:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760768.922037:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.922047:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.922076:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760768.922108:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493803904:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760768.922198:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6401, 6401] 00000080:00200000:0.0:1789760768.922217:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.922223:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.922238:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26222592 i_blocks 53255 00000020:00200000:0.0:1789760768.922242:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.922253:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.922287:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26222592, count: 4096 00000080:00200000:0.0:1789760768.922294:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26222592, bytes: 4096 00000080:00200000:0.0:1789760768.922308:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.922317:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.922319:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26222592, 26226688) 0 0 00000080:00200000:0.0:1789760768.922324:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.922340:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6402, 6402] 00020000:00200000:0.0:1789760768.922358:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26222592 chunk: [26222592, 26226688) 26226688, 4096 00020000:00200000:0.0:1789760768.922364:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.922383:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6402, 6402] 00000008:00010000:0.0:1789760768.922431:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.922438:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760768.922459:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26222592, 26226688) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.922474:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26222592=0x1902000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.922525:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.922528:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26226688 00000020:00010000:0.0:1789760768.922536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.922539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.922542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.922546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.922549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.922557:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.922564:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.923425:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:1.0:1789760768.923547:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26222592<>26226688 00000100:00100000:1.0:1789760768.924133:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760768.924162:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493804032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760769.018593:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973721 gen 3 00000008:00010000:1.0:1789760769.018697:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760769.018706:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.018717:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.018743:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760769.018772:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493804032:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.018885:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6402, 6402] 00000080:00200000:0.0:1789760769.018901:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.018908:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.018923:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26226688 i_blocks 53255 00000020:00200000:0.0:1789760769.018928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.018938:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.018971:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26226688, count: 4096 00000080:00200000:0.0:1789760769.018979:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26226688, bytes: 4096 00000080:00200000:0.0:1789760769.018987:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.018997:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.019000:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26226688, 26230784) 0 0 00000080:00200000:0.0:1789760769.019005:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.019024:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6403, 6403] 00020000:00200000:0.0:1789760769.019032:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26226688 chunk: [26226688, 26230784) 26230784, 4096 00020000:00200000:0.0:1789760769.019037:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.019057:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6403, 6403] 00000008:00010000:0.0:1789760769.019076:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.019082:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.019093:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26226688, 26230784) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.019109:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26226688=0x1903000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.019171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.019174:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26230784 00000020:00010000:0.0:1789760769.019181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.019184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.019188:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.019191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.019193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.019201:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.019208:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.019223:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760769.019315:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 26226688<>26230784 00000100:00100000:0.0:1789760769.019352:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760769.019367:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804160:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760769.054434:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760769.054450:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.054460:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.054486:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760769.054517:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760769.054662:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6403, 6403] 00000080:00200000:2.0:1789760769.054677:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.054684:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.054698:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26230784 i_blocks 53255 00000020:00200000:2.0:1789760769.054702:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.054712:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.054743:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26230784, count: 4096 00000080:00200000:2.0:1789760769.054750:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26230784, bytes: 4096 00000080:00200000:2.0:1789760769.054759:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.054768:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.054770:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26230784, 26234880) 0 0 00000080:00200000:2.0:1789760769.054774:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.054788:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6404, 6404] 00020000:00200000:2.0:1789760769.054796:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26230784 chunk: [26230784, 26234880) 26234880, 4096 00020000:00200000:2.0:1789760769.054800:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.054827:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6404, 6404] 00000008:00010000:2.0:1789760769.054843:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.054849:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760769.054861:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26230784, 26234880) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.054874:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26230784=0x1904000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.054926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.054928:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26234880 00000020:00010000:2.0:1789760769.054936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.054940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.054943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.054946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.054949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.054956:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.054962:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.054976:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760769.055066:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26230784<>26234880 00000100:00100000:3.0:1789760769.055140:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493804288:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.070467:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1789760769.070487:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.070520:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760769.086378:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973723 gen 3 00000008:00010000:3.0:1789760769.086468:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760769.086477:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.086488:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.086506:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760769.086532:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493804288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760769.087180:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6404, 6404] 00000080:00200000:2.0:1789760769.087199:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.087205:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.087221:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26234880 i_blocks 53255 00000020:00200000:2.0:1789760769.087226:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.087237:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.087269:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26234880, count: 4096 00000080:00200000:2.0:1789760769.087276:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26234880, bytes: 4096 00000080:00200000:2.0:1789760769.087288:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.087296:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.087298:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26234880, 26238976) 0 0 00000080:00200000:2.0:1789760769.087303:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.087320:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6405, 6405] 00020000:00200000:2.0:1789760769.087327:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26234880 chunk: [26234880, 26238976) 26238976, 4096 00020000:00200000:2.0:1789760769.087332:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.087350:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6405, 6405] 00000008:00010000:2.0:1789760769.087366:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.087372:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760769.087382:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26234880, 26238976) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.087421:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26234880=0x1905000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.087478:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.087480:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26238976 00000020:00010000:2.0:1789760769.087488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.087492:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.087495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.087499:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.087502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.087509:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.087515:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.087530:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760769.087707:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26234880<>26238976 00000100:00100000:3.0:1789760769.087907:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493804416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760769.118177:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973724 gen 3 00000008:00010000:3.0:1789760769.118238:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760769.118246:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.118256:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.118275:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760769.118302:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493804416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760769.118515:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6405, 6405] 00000080:00200000:2.0:1789760769.118530:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760769.118536:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760769.118551:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26238976 i_blocks 53255 00000020:00200000:2.0:1789760769.118555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760769.118565:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760769.118597:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26238976, count: 4096 00000080:00200000:2.0:1789760769.118613:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26238976, bytes: 4096 00000080:00200000:2.0:1789760769.118623:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760769.118633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760769.118635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26238976, 26243072) 0 0 00000080:00200000:2.0:1789760769.118640:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760769.118656:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6406, 6406] 00020000:00200000:2.0:1789760769.118664:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26238976 chunk: [26238976, 26243072) 26243072, 4096 00020000:00200000:2.0:1789760769.118668:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760769.118687:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6406, 6406] 00000008:00010000:2.0:1789760769.118705:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760769.118711:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760769.118724:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26238976, 26243072) append 0 append lockpos 0 00000080:00200000:2.0:1789760769.118739:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26238976=0x1906000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760769.118801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760769.118804:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26243072 00000020:00010000:2.0:1789760769.118812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760769.118828:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.118831:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760769.118835:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760769.118838:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760769.118846:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760769.118854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760769.118871:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760769.119154:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26238976<>26243072 00000100:00100000:3.0:1789760769.119278:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493804544:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760769.153931:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973725 gen 3 00000008:00010000:3.0:1789760769.153985:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760769.153994:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.154006:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.154032:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760769.154063:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493804544:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.154186:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6406, 6406] 00000080:00200000:0.0:1789760769.154202:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.154210:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.154227:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26243072 i_blocks 53255 00000020:00200000:0.0:1789760769.154231:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.154241:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.154277:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26243072, count: 4096 00000080:00200000:0.0:1789760769.154285:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26243072, bytes: 4096 00000080:00200000:0.0:1789760769.154296:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.154305:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.154308:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26243072, 26247168) 0 0 00000080:00200000:0.0:1789760769.154313:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.154330:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6407, 6407] 00020000:00200000:0.0:1789760769.154338:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26243072 chunk: [26243072, 26247168) 26247168, 4096 00020000:00200000:0.0:1789760769.154343:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.154363:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6407, 6407] 00000008:00010000:0.0:1789760769.154381:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.154388:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.154463:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26243072, 26247168) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.154478:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26243072=0x1907000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.154533:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.154536:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26247168 00000020:00010000:0.0:1789760769.154545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.154548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.154552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.154555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.154558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.154565:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.154572:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.154584:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760769.154691:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 26243072<>26247168 00000100:00100000:1.0:1789760769.154860:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760769.205771:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973726 gen 3 00000008:00010000:1.0:1789760769.205827:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760769.205835:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.205845:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.205864:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760769.205890:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804672:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.206107:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6407, 6407] 00000080:00200000:0.0:1789760769.206134:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.206139:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.206155:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26247168 i_blocks 53255 00000020:00200000:0.0:1789760769.206158:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.206168:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.206212:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26247168, count: 4096 00000080:00200000:0.0:1789760769.206219:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26247168, bytes: 4096 00000080:00200000:0.0:1789760769.206229:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.206239:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.206241:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26247168, 26251264) 0 0 00000080:00200000:0.0:1789760769.206246:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.206261:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6408, 6408] 00020000:00200000:0.0:1789760769.206269:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26247168 chunk: [26247168, 26251264) 26251264, 4096 00020000:00200000:0.0:1789760769.206274:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.206291:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6408, 6408] 00000008:00010000:0.0:1789760769.206309:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.206315:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.206326:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26247168, 26251264) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.206338:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26247168=0x1908000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.206434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.206437:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26251264 00000020:00010000:0.0:1789760769.206446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.206449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.206452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.206456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.206458:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.206465:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.206473:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.206489:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760769.206589:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 26247168<>26251264 00000100:00100000:1.0:1789760769.206781:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.206799:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804800:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760769.260624:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760769.290517:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973727 gen 3 00000008:00010000:1.0:1789760769.290577:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760769.290585:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.290597:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.290667:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760769.290697:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804800:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.293446:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6408, 6408] 00000080:00200000:0.0:1789760769.293473:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.293479:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.293497:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26251264 i_blocks 53255 00000020:00200000:0.0:1789760769.293502:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.293529:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.293563:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26251264, count: 4096 00000080:00200000:0.0:1789760769.293571:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26251264, bytes: 4096 00000080:00200000:0.0:1789760769.293682:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.293692:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.293694:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26251264, 26255360) 0 0 00000080:00200000:0.0:1789760769.293699:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.293715:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6409, 6409] 00020000:00200000:0.0:1789760769.293723:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26251264 chunk: [26251264, 26255360) 26255360, 4096 00020000:00200000:0.0:1789760769.293727:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.293748:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6409, 6409] 00000008:00010000:0.0:1789760769.293766:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.293772:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.293782:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26251264, 26255360) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.293796:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26251264=0x1909000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.293856:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.293859:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26255360 00000020:00010000:0.0:1789760769.293866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.293870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.293873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.293877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.293880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.293887:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.293893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.293917:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760769.294160:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 26251264<>26255360 00000100:00100000:1.0:1789760769.294253:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760769.354330:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973728 gen 3 00000008:00010000:1.0:1789760769.354366:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760769.354372:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.354381:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.354506:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760769.354532:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493804928:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.358306:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6409, 6409] 00000080:00200000:0.0:1789760769.358326:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.358333:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.358348:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26255360 i_blocks 53255 00000020:00200000:0.0:1789760769.358353:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.358367:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.358413:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26255360, count: 4096 00000080:00200000:0.0:1789760769.358421:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26255360, bytes: 4096 00000080:00200000:0.0:1789760769.359037:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.359047:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.359049:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26255360, 26259456) 0 0 00000080:00200000:0.0:1789760769.359054:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.359068:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6410, 6410] 00020000:00200000:0.0:1789760769.359076:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26255360 chunk: [26255360, 26259456) 26259456, 4096 00020000:00200000:0.0:1789760769.359080:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.359097:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6410, 6410] 00000008:00010000:0.0:1789760769.359239:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.359245:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.359255:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26255360, 26259456) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.359266:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26255360=0x190a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.359327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.359330:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26259456 00000020:00010000:0.0:1789760769.359337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.359341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.359343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.359346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.359349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.359356:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.359361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.359388:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760769.359563:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 26255360<>26259456 00000100:00100000:1.0:1789760769.359644:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805056:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760769.408102:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760769.408238:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.408251:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.408322:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760769.408351:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805056:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.408504:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6410, 6410] 00000080:00200000:0.0:1789760769.408520:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.408526:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.408675:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26259456 i_blocks 53255 00000020:00200000:1.0:1789760769.408680:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.408691:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.408724:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26259456, count: 4096 00000080:00200000:1.0:1789760769.408732:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26259456, bytes: 4096 00000080:00200000:1.0:1789760769.408744:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.408753:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.408755:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26259456, 26263552) 0 0 00000080:00200000:1.0:1789760769.408760:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.408776:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6411, 6411] 00020000:00200000:1.0:1789760769.408785:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26259456 chunk: [26259456, 26263552) 26263552, 4096 00020000:00200000:1.0:1789760769.408789:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.408807:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6411, 6411] 00000008:00010000:1.0:1789760769.408825:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.408831:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.408845:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26259456, 26263552) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.408858:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26259456=0x190b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.409071:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.409074:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26263552 00000020:00010000:1.0:1789760769.409083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.409086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.409090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.409093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.409096:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.409107:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.409115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.409171:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760769.409280:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26259456<>26263552 00000100:00100000:1.0:1789760769.409323:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760769.409340:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805184:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760769.474318:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973730 gen 3 00000008:00010000:1.0:1789760769.474381:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:1.0:1789760769.474389:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.474433:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.474449:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:1.0:1789760769.474476:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805184:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.474580:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6411, 6411] 00000080:00200000:0.0:1789760769.474597:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.474612:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.474807:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26263552 i_blocks 53255 00000020:00200000:0.0:1789760769.474812:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.474823:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.474853:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26263552, count: 4096 00000080:00200000:0.0:1789760769.474861:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26263552, bytes: 4096 00000080:00200000:0.0:1789760769.474871:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.474880:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.474882:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26263552, 26267648) 0 0 00000080:00200000:0.0:1789760769.474887:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.474904:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6412, 6412] 00020000:00200000:0.0:1789760769.474913:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26263552 chunk: [26263552, 26267648) 26267648, 4096 00020000:00200000:0.0:1789760769.474917:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.474938:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6412, 6412] 00000008:00010000:0.0:1789760769.475032:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.475039:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.475052:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26263552, 26267648) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.475071:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26263552=0x190c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.475140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.475142:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26267648 00000020:00010000:0.0:1789760769.475151:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.475154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.475157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.475160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.475164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.475171:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.475179:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.475195:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760769.475298:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 26263552<>26267648 00000100:00100000:0.0:1789760769.475342:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760769.475361:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805312:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760769.545459:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760769.545474:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.545486:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.545515:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760769.545555:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805312:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.545653:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6412, 6412] 00000080:00200000:1.0:1789760769.545670:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.545677:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.545694:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26267648 i_blocks 53255 00000020:00200000:1.0:1789760769.545698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.545709:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.545744:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26267648, count: 4096 00000080:00200000:1.0:1789760769.545753:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26267648, bytes: 4096 00000080:00200000:1.0:1789760769.545763:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.545773:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.545776:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26267648, 26271744) 0 0 00000080:00200000:1.0:1789760769.545781:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.545800:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6413, 6413] 00020000:00200000:1.0:1789760769.545810:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26267648 chunk: [26267648, 26271744) 26271744, 4096 00020000:00200000:1.0:1789760769.545815:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.545836:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6413, 6413] 00000008:00010000:1.0:1789760769.545857:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.545863:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.545875:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26267648, 26271744) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.545889:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26267648=0x190d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.545951:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.545954:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26271744 00000020:00010000:1.0:1789760769.545963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.545966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.545968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.545972:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.545975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.545985:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.545993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.546008:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760769.546141:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26267648<>26271744 00000100:00100000:1.0:1789760769.546180:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760769.546196:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805440:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760769.583454:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973732 gen 3 00000008:00010000:1.0:1789760769.583518:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:1.0:1789760769.583526:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.583535:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.583561:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:1.0:1789760769.583588:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805440:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760769.583688:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6413, 6413] 00000080:00200000:0.0:1789760769.583704:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.583712:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.583729:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26271744 i_blocks 53255 00000020:00200000:0.0:1789760769.583733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.583743:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.583779:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26271744, count: 4096 00000080:00200000:0.0:1789760769.583785:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26271744, bytes: 4096 00000080:00200000:0.0:1789760769.583795:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.583803:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.583805:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26271744, 26275840) 0 0 00000080:00200000:0.0:1789760769.583809:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.583904:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6414, 6414] 00020000:00200000:0.0:1789760769.583914:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26271744 chunk: [26271744, 26275840) 26275840, 4096 00020000:00200000:0.0:1789760769.583918:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.583939:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6414, 6414] 00000008:00010000:0.0:1789760769.583956:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.583961:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760769.583971:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26271744, 26275840) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.583982:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26271744=0x190e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.584032:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.584034:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26275840 00000020:00010000:0.0:1789760769.584040:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.584043:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.584045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.584048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.584050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.584055:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.584060:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.584073:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760769.584388:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 26271744<>26275840 00000100:00100000:0.0:1789760769.584645:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760769.584666:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805568:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760769.642068:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760769.642086:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.642099:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.642728:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760769.642775:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805568:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.645077:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6414, 6414] 00000080:00200000:1.0:1789760769.646350:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.646363:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.646384:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26275840 i_blocks 53255 00000020:00200000:1.0:1789760769.646390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.646422:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.646460:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26275840, count: 4096 00000080:00200000:1.0:1789760769.646469:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26275840, bytes: 4096 00000080:00200000:1.0:1789760769.646484:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.646493:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.646496:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26275840, 26279936) 0 0 00000080:00200000:1.0:1789760769.646500:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.646517:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6415, 6415] 00020000:00200000:1.0:1789760769.646526:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26275840 chunk: [26275840, 26279936) 26279936, 4096 00020000:00200000:1.0:1789760769.646531:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.646551:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6415, 6415] 00000008:00010000:1.0:1789760769.646572:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.646579:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.646592:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26275840, 26279936) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.646608:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26275840=0x190f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.646679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.646683:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26279936 00000020:00010000:1.0:1789760769.646691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.646694:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.646697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.646700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.646703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.646717:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.646725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.646754:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760769.646874:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 26275840<>26279936 00000100:00100000:0.0:1789760769.648447:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805696:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760769.748681:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973734 gen 3 00000008:00010000:0.0:1789760769.748754:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760769.748762:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.748771:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.748799:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760769.748828:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805696:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.750213:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6415, 6415] 00000080:00200000:1.0:1789760769.750234:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.750241:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.750258:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26279936 i_blocks 53255 00000020:00200000:1.0:1789760769.750263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.750277:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.750312:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26279936, count: 4096 00000080:00200000:1.0:1789760769.750320:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26279936, bytes: 4096 00000080:00200000:1.0:1789760769.750333:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.750342:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.750345:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26279936, 26284032) 0 0 00000080:00200000:1.0:1789760769.750349:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.750368:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6416, 6416] 00020000:00200000:1.0:1789760769.750378:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26279936 chunk: [26279936, 26284032) 26284032, 4096 00020000:00200000:1.0:1789760769.750383:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.750436:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6416, 6416] 00000008:00010000:1.0:1789760769.750455:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.750461:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.750473:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26279936, 26284032) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.750487:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26279936=0x1910000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.750559:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.750562:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26284032 00000020:00010000:1.0:1789760769.750571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.750575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.750578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.750582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.750585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.750598:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.750605:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.750624:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760769.750734:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 26279936<>26284032 00000100:00100000:0.0:1789760769.751652:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805824:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760769.788598:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760769.788623:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.788634:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.788659:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760769.788690:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493805824:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.788853:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6416, 6416] 00000080:00200000:1.0:1789760769.788869:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.788874:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.788889:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26284032 i_blocks 53255 00000020:00200000:1.0:1789760769.788892:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.788902:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.788934:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26284032, count: 4096 00000080:00200000:1.0:1789760769.788942:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26284032, bytes: 4096 00000080:00200000:1.0:1789760769.788954:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.788962:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.788964:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26284032, 26288128) 0 0 00000080:00200000:1.0:1789760769.788969:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.788985:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6417, 6417] 00020000:00200000:1.0:1789760769.788994:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26284032 chunk: [26284032, 26288128) 26288128, 4096 00020000:00200000:1.0:1789760769.789000:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.789020:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6417, 6417] 00000008:00010000:1.0:1789760769.789040:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.789047:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.789060:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26284032, 26288128) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.789073:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26284032=0x1911000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.789369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.789373:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26288128 00000020:00010000:1.0:1789760769.789382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.789386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.789389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.789487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.789491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.789503:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.789511:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.789528:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92c00 00000008:00100000:1.0:1789760769.789901:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 26284032<>26288128 00000100:00100000:0.0:1789760769.789972:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805952:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760769.833446:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973736 gen 3 00000008:00010000:0.0:1789760769.833514:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92c00 00000020:00200000:0.0:1789760769.833522:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.833533:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.833552:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92c00 00000100:00100000:0.0:1789760769.833581:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493805952:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760769.833754:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6417, 6417] 00000080:00200000:3.0:1789760769.833771:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.833778:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.833794:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26288128 i_blocks 53255 00000020:00200000:3.0:1789760769.833798:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.833809:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.833854:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26288128, count: 4096 00000080:00200000:3.0:1789760769.833862:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26288128, bytes: 4096 00000080:00200000:3.0:1789760769.833871:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.833880:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.833882:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26288128, 26292224) 0 0 00000080:00200000:3.0:1789760769.833887:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.833905:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6418, 6418] 00020000:00200000:3.0:1789760769.833913:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26288128 chunk: [26288128, 26292224) 26292224, 4096 00020000:00200000:3.0:1789760769.833918:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.833937:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6418, 6418] 00000008:00010000:3.0:1789760769.833957:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.833963:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760769.833976:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26288128, 26292224) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.833990:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26288128=0x1912000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.834047:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.834050:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26292224 00000020:00010000:3.0:1789760769.834058:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.834061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.834064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.834067:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.834070:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.834077:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.834083:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.834098:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760769.834231:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 26288128<>26292224 00000100:00100000:3.0:1789760769.834280:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.834298:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493806080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760769.913520:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973737 gen 3 00000008:00010000:3.0:1789760769.913569:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760769.913579:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.913591:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.913625:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760769.913654:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493806080:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.913858:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6418, 6418] 00000080:00200000:1.0:1789760769.913875:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.913881:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.913897:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26292224 i_blocks 53255 00000020:00200000:1.0:1789760769.913901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.913909:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.913944:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26292224, count: 4096 00000080:00200000:1.0:1789760769.913951:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26292224, bytes: 4096 00000080:00200000:1.0:1789760769.913961:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.913969:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.913971:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26292224, 26296320) 0 0 00000080:00200000:1.0:1789760769.913975:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.913990:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6419, 6419] 00020000:00200000:1.0:1789760769.913999:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26292224 chunk: [26292224, 26296320) 26296320, 4096 00020000:00200000:1.0:1789760769.914003:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.914020:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6419, 6419] 00000008:00010000:1.0:1789760769.914036:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.914042:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.914053:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26292224, 26296320) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.914067:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26292224=0x1913000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.914133:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.914136:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26296320 00000020:00010000:1.0:1789760769.914145:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.914148:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.914152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.914164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.914167:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.914184:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.914191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.914208:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92180 00000008:00100000:1.0:1789760769.914312:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 26292224<>26296320 00000100:00100000:0.0:1789760769.915987:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806208:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760769.969004:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973738 gen 3 00000008:00010000:0.0:1789760769.969064:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92180 00000020:00200000:0.0:1789760769.969073:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.969084:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.969110:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92180 00000100:00100000:0.0:1789760769.969148:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806208:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760769.969471:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6419, 6419] 00000080:00200000:1.0:1789760769.969488:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.969495:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.969511:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26296320 i_blocks 53255 00000020:00200000:1.0:1789760769.969514:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.969525:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.969561:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26296320, count: 4096 00000080:00200000:1.0:1789760769.969569:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26296320, bytes: 4096 00000080:00200000:1.0:1789760769.969579:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.969588:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.969590:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26296320, 26300416) 0 0 00000080:00200000:1.0:1789760769.969595:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.969613:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6420, 6420] 00020000:00200000:1.0:1789760769.969622:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26296320 chunk: [26296320, 26300416) 26300416, 4096 00020000:00200000:1.0:1789760769.969626:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.969648:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6420, 6420] 00000008:00010000:1.0:1789760769.969666:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.969671:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760769.969683:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26296320, 26300416) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.969697:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26296320=0x1914000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.969752:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.969755:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26300416 00000020:00010000:1.0:1789760769.969763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.969767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.969770:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.969773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.969776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.969787:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.969794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.969808:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760769.969909:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 26296320<>26300416 00000100:00100000:0.0:1789760769.970066:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760769.970086:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806336:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760770.032239:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973739 gen 3 00000008:00010000:0.0:1789760770.032294:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760770.032302:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.032314:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.032450:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760770.032476:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806336:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.035964:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6420, 6420] 00000080:00200000:1.0:1789760770.035990:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.035996:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.036014:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26300416 i_blocks 53255 00000020:00200000:1.0:1789760770.036019:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.036035:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.036068:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26300416, count: 4096 00000080:00200000:1.0:1789760770.036076:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26300416, bytes: 4096 00000080:00200000:1.0:1789760770.036089:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.036099:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.036102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26300416, 26304512) 0 0 00000080:00200000:1.0:1789760770.036107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.036133:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6421, 6421] 00020000:00200000:1.0:1789760770.036142:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26300416 chunk: [26300416, 26304512) 26304512, 4096 00020000:00200000:1.0:1789760770.036146:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.036166:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6421, 6421] 00000008:00010000:1.0:1789760770.036183:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.036188:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.036198:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26300416, 26304512) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.036214:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26300416=0x1915000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.036274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.036277:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26304512 00000020:00010000:1.0:1789760770.036285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.036288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.036292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.036295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.036298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.036310:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.036317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.036342:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760770.036482:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 26300416<>26304512 00000100:00100000:0.0:1789760770.036656:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806464:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760770.076844:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760770.076861:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.076872:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.077006:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760770.077038:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806464:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.077100:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6421, 6421] 00000080:00200000:1.0:1789760770.077133:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.077139:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.077162:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26304512 i_blocks 53255 00000020:00200000:1.0:1789760770.077166:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.077183:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.077219:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26304512, count: 4096 00000080:00200000:1.0:1789760770.077227:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26304512, bytes: 4096 00000080:00200000:1.0:1789760770.077237:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.077246:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.077249:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26304512, 26308608) 0 0 00000080:00200000:1.0:1789760770.077253:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.077271:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6422, 6422] 00020000:00200000:1.0:1789760770.077280:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26304512 chunk: [26304512, 26308608) 26308608, 4096 00020000:00200000:1.0:1789760770.077285:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.077305:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6422, 6422] 00000008:00010000:1.0:1789760770.077324:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.077330:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.077344:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26304512, 26308608) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.077357:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26304512=0x1916000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.077441:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.077444:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26308608 00000020:00010000:1.0:1789760770.077452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.077456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.077460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.077463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.077466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.077477:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.077485:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.077501:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760770.077604:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 26304512<>26308608 00000100:00100000:0.0:1789760770.077668:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493806592:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760770.133005:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973740 gen 3 00000100:00100000:0.0:1789760770.133049:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016dc00 x1876699493806592/t4294973741(4294973741) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760770.133101:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760770.133109:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.133130:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.133172:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000080:00200000:1.0:1789760770.133244:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6422, 6422] 00000080:00200000:1.0:1789760770.133262:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1789760770.133266:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493806592:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.133268:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.133282:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26308608 i_blocks 53255 00000020:00200000:1.0:1789760770.133286:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.133295:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.133329:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26308608, count: 4096 00000080:00200000:1.0:1789760770.133337:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26308608, bytes: 4096 00000080:00200000:1.0:1789760770.133348:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.133355:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.133357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26308608, 26312704) 0 0 00000080:00200000:1.0:1789760770.133361:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.133375:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6423, 6423] 00020000:00200000:1.0:1789760770.133382:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26308608 chunk: [26308608, 26312704) 26312704, 4096 00020000:00200000:1.0:1789760770.133386:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.133431:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6423, 6423] 00000008:00010000:1.0:1789760770.133447:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.133452:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.133462:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26308608, 26312704) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.133477:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26308608=0x1917000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.133531:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.133533:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26312704 00000020:00010000:1.0:1789760770.133540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.133542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.133545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.133547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.133549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.133558:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.133563:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.133575:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760770.133734:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 26308608<>26312704 00000100:00100000:0.0:1789760770.134032:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806720:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760770.182019:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973742 gen 3 00000008:00010000:0.0:1789760770.182083:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760770.182091:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.182101:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.182131:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760770.182162:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806720:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.182490:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6423, 6423] 00000080:00200000:1.0:1789760770.182505:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.182511:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.182526:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26312704 i_blocks 53255 00000020:00200000:1.0:1789760770.182530:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.182540:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.182572:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26312704, count: 4096 00000080:00200000:1.0:1789760770.182580:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26312704, bytes: 4096 00000080:00200000:1.0:1789760770.182590:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.182609:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.182611:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26312704, 26316800) 0 0 00000080:00200000:1.0:1789760770.182616:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.182702:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6424, 6424] 00020000:00200000:1.0:1789760770.182711:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26312704 chunk: [26312704, 26316800) 26316800, 4096 00020000:00200000:1.0:1789760770.182716:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.182736:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6424, 6424] 00000008:00010000:1.0:1789760770.182753:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.182760:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.182772:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26312704, 26316800) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.182786:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26312704=0x1918000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.182853:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.182856:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26316800 00000020:00010000:1.0:1789760770.182864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.182868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.182871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.182945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.182949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.182963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.182971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.182989:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760770.183093:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 26312704<>26316800 00000100:00100000:0.0:1789760770.183181:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493806848:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760770.215774:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973743 gen 3 00000008:00010000:0.0:1789760770.215924:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760770.215934:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.215945:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.215968:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760770.215998:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493806848:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.216014:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6424, 6424] 00000080:00200000:1.0:1789760770.216032:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.216038:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.216054:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26316800 i_blocks 53255 00000020:00200000:1.0:1789760770.216058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.216067:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.216099:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26316800, count: 4096 00000080:00200000:1.0:1789760770.216106:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26316800, bytes: 4096 00000080:00200000:1.0:1789760770.216125:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.216135:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.216137:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26316800, 26320896) 0 0 00000080:00200000:1.0:1789760770.216142:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.216158:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6425, 6425] 00020000:00200000:1.0:1789760770.216167:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26316800 chunk: [26316800, 26320896) 26320896, 4096 00020000:00200000:1.0:1789760770.216171:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.216266:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6425, 6425] 00000008:00010000:1.0:1789760770.216285:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.216291:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.216303:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26316800, 26320896) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.216317:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26316800=0x1919000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.216374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.216377:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26320896 00000020:00010000:1.0:1789760770.216385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.216389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.216432:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.216438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.216441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.216453:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.216462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.216477:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760770.216574:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 26316800<>26320896 00000100:00100000:0.0:1789760770.216670:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806976:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760770.255992:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760770.256009:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.256019:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.256079:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760770.256110:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493806976:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760770.256863:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6425, 6425] 00000080:00200000:1.0:1789760770.256949:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.256957:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.256973:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26320896 i_blocks 53255 00000020:00200000:1.0:1789760770.256978:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.256989:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.257022:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26320896, count: 4096 00000080:00200000:1.0:1789760770.257030:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26320896, bytes: 4096 00000080:00200000:1.0:1789760770.257042:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.257051:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.257054:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26320896, 26324992) 0 0 00000080:00200000:1.0:1789760770.257059:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.257074:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6426, 6426] 00020000:00200000:1.0:1789760770.257083:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26320896 chunk: [26320896, 26324992) 26324992, 4096 00020000:00200000:1.0:1789760770.257088:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.257106:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6426, 6426] 00000008:00010000:1.0:1789760770.259125:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.259140:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760770.259159:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26320896, 26324992) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.259179:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26320896=0x191a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.259243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.259245:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26324992 00000020:00010000:1.0:1789760770.259253:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.259255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.259258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.259260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.259262:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.259274:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.259282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.259307:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760770.259603:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 26320896<>26324992 00000100:00100000:0.0:1789760770.259685:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493807104:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760770.286017:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760770.290377:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973745 gen 3 00000008:00010000:0.0:1789760770.290469:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760770.290479:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.290489:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.290513:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760770.290542:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493807104:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760770.290646:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6426, 6426] 00000080:00200000:3.0:1789760770.290663:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.290668:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.290684:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26324992 i_blocks 53255 00000020:00200000:3.0:1789760770.290688:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.290698:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.290730:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26324992, count: 4096 00000080:00200000:3.0:1789760770.290738:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26324992, bytes: 4096 00000080:00200000:3.0:1789760770.290756:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.290766:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.290769:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26324992, 26329088) 0 0 00000080:00200000:3.0:1789760770.290773:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.290790:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6427, 6427] 00020000:00200000:3.0:1789760770.290799:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26324992 chunk: [26324992, 26329088) 26329088, 4096 00020000:00200000:3.0:1789760770.290808:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.290830:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6427, 6427] 00000008:00010000:3.0:1789760770.290848:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.290854:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760770.290866:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26324992, 26329088) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.290881:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26324992=0x191b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.290938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.290941:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26329088 00000020:00010000:3.0:1789760770.290951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.290955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.290958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.290961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.290964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.290973:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.290980:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.290995:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760770.291098:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 26324992<>26329088 00000100:00100000:2.0:1789760770.292530:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760770.292562:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493807232:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760770.332238:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760770.332254:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.332265:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.332289:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760770.332320:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493807232:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760770.332424:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6427, 6427] 00000080:00200000:0.0:1789760770.332440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.332447:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.332464:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26329088 i_blocks 53255 00000020:00200000:0.0:1789760770.332467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.332477:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.332512:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26329088, count: 4096 00000080:00200000:0.0:1789760770.332519:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26329088, bytes: 4096 00000080:00200000:0.0:1789760770.332529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.332538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.332541:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26329088, 26333184) 0 0 00000080:00200000:0.0:1789760770.332546:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.332562:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6428, 6428] 00020000:00200000:0.0:1789760770.332571:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26329088 chunk: [26329088, 26333184) 26333184, 4096 00020000:00200000:0.0:1789760770.332575:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.332594:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6428, 6428] 00000008:00010000:0.0:1789760770.332614:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.332620:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760770.332631:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26329088, 26333184) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.332644:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26329088=0x191c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.332699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.332703:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26333184 00000020:00010000:0.0:1789760770.332711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.332715:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.332718:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.332722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.332725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.332814:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.332821:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.332838:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760770.332936:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 26329088<>26333184 00000100:00100000:1.0:1789760770.333021:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493807360:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760770.378547:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973746 gen 3 00000100:00100000:0.0:1789760770.378590:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304e300 x1876699493807360/t4294973747(4294973747) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760770.378640:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760770.378648:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.378658:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.378802:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760770.378829:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493807360:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760770.380870:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6428, 6428] 00000080:00200000:3.0:1789760770.380898:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.380906:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.380923:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26333184 i_blocks 53255 00000020:00200000:3.0:1789760770.380929:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.380946:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.380980:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26333184, count: 4096 00000080:00200000:3.0:1789760770.380988:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26333184, bytes: 4096 00000080:00200000:3.0:1789760770.381003:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.381012:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.381014:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26333184, 26337280) 0 0 00000080:00200000:3.0:1789760770.381020:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.381037:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6429, 6429] 00020000:00200000:3.0:1789760770.381046:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26333184 chunk: [26333184, 26337280) 26337280, 4096 00020000:00200000:3.0:1789760770.381050:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.381072:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6429, 6429] 00000008:00010000:3.0:1789760770.381091:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.381096:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760770.381110:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26333184, 26337280) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.381134:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26333184=0x191d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.381204:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.381208:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26337280 00000020:00010000:3.0:1789760770.381216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.381220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.381223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.381227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.381230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.381239:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.381246:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.381271:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760770.381380:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 26333184<>26337280 00000100:00100000:2.0:1789760770.381623:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760770.385054:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.426801:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973747 gen 3 00000100:00100000:3.0:1789760770.426857:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b480 x1876699493807488/t4294973748(4294973748) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760770.426913:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760770.426921:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.426930:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.427008:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760770.427033:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807488:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.431914:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6429, 6429] 00000080:00200000:2.0:1789760770.431944:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.431952:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.431971:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26337280 i_blocks 53255 00000020:00200000:2.0:1789760770.431978:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.431996:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.432032:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26337280, count: 4096 00000080:00200000:2.0:1789760770.432041:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26337280, bytes: 4096 00000080:00200000:2.0:1789760770.432057:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.432068:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.432071:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26337280, 26341376) 0 0 00000080:00200000:2.0:1789760770.432075:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.432093:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6430, 6430] 00020000:00200000:2.0:1789760770.432103:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26337280 chunk: [26337280, 26341376) 26341376, 4096 00020000:00200000:2.0:1789760770.432108:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.433824:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6430, 6430] 00000008:00010000:2.0:1789760770.433863:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.433870:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.433885:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26337280, 26341376) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.433902:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26337280=0x191e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.433979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.433984:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26341376 00000020:00010000:2.0:1789760770.433992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.433996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.433999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.434003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.434007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.434018:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.434027:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.434056:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760770.434187:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26337280<>26341376 00000100:00100000:3.0:1789760770.434370:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.491066:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973749 gen 3 00000008:00010000:3.0:1789760770.491145:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760770.491155:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.491165:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.491190:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760770.491220:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807616:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760770.491855:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6430, 6430] 00000080:00200000:0.0:1789760770.491873:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.491880:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.491897:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26341376 i_blocks 53255 00000020:00200000:0.0:1789760770.491901:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.491913:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.491950:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26341376, count: 4096 00000080:00200000:0.0:1789760770.491958:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26341376, bytes: 4096 00000080:00200000:0.0:1789760770.491969:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.491980:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.491982:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26341376, 26345472) 0 0 00000080:00200000:0.0:1789760770.491988:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.492006:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6431, 6431] 00020000:00200000:0.0:1789760770.492015:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26341376 chunk: [26341376, 26345472) 26345472, 4096 00020000:00200000:0.0:1789760770.492020:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.492041:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6431, 6431] 00000008:00010000:0.0:1789760770.492061:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.492068:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760770.492080:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26341376, 26345472) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.492095:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26341376=0x191f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.492168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.492171:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26345472 00000020:00010000:0.0:1789760770.492180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.492184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.492187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.492191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.492194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.492201:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.492208:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.492224:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760770.492333:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 26341376<>26345472 00000100:00100000:1.0:1789760770.492482:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493807744:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760770.528280:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973750 gen 3 00000008:00010000:1.0:1789760770.528333:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760770.528341:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.528351:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.530211:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760770.530260:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493807744:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760770.531250:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6431, 6431] 00000080:00200000:3.0:1789760770.531273:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.531281:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.531302:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26345472 i_blocks 53255 00000020:00200000:3.0:1789760770.531307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.531321:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.531356:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26345472, count: 4096 00000080:00200000:3.0:1789760770.531364:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26345472, bytes: 4096 00000080:00200000:3.0:1789760770.531378:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.531388:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.531390:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26345472, 26349568) 0 0 00000080:00200000:3.0:1789760770.531417:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.531434:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6432, 6432] 00020000:00200000:3.0:1789760770.531444:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26345472 chunk: [26345472, 26349568) 26349568, 4096 00020000:00200000:3.0:1789760770.531448:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.531470:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6432, 6432] 00000008:00010000:3.0:1789760770.531489:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.531496:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760770.531508:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26345472, 26349568) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.531524:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26345472=0x1920000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.531595:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.531598:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26349568 00000020:00010000:3.0:1789760770.531606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.531610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.531614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.531617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.531621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.531630:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.531637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.531658:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760770.531773:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 26345472<>26349568 00000100:00100000:3.0:1789760770.531829:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.531854:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807872:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760770.578291:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760770.578308:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.578320:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.578341:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760770.578372:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493807872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.578495:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6432, 6432] 00000080:00200000:2.0:1789760770.578513:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.578519:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.578536:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26349568 i_blocks 53255 00000020:00200000:2.0:1789760770.578541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.578551:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.578584:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26349568, count: 4096 00000080:00200000:2.0:1789760770.578603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26349568, bytes: 4096 00000080:00200000:2.0:1789760770.578614:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.578623:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.578626:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26349568, 26353664) 0 0 00000080:00200000:2.0:1789760770.578631:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.578648:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6433, 6433] 00020000:00200000:2.0:1789760770.578658:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26349568 chunk: [26349568, 26353664) 26353664, 4096 00020000:00200000:2.0:1789760770.578662:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.578684:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6433, 6433] 00000008:00010000:2.0:1789760770.578703:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.578709:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.578722:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26349568, 26353664) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.578738:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26349568=0x1921000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.578802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.578805:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26353664 00000020:00010000:2.0:1789760770.578814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.578818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.578821:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.578825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.578829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.578837:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.578844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.578861:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760770.578967:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26349568<>26353664 00000100:00100000:2.0:1789760770.579011:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760770.579029:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.623831:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973752 gen 3 00000008:00010000:3.0:1789760770.623896:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760770.623904:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.623915:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.624019:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760770.624050:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.626552:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6433, 6433] 00000080:00200000:2.0:1789760770.626580:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.626587:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.626606:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26353664 i_blocks 53255 00000020:00200000:2.0:1789760770.626611:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.626625:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.626660:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26353664, count: 4096 00000080:00200000:2.0:1789760770.626669:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26353664, bytes: 4096 00000080:00200000:2.0:1789760770.626683:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.626692:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.626694:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26353664, 26357760) 0 0 00000080:00200000:2.0:1789760770.626699:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.626715:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6434, 6434] 00020000:00200000:2.0:1789760770.626724:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26353664 chunk: [26353664, 26357760) 26357760, 4096 00020000:00200000:2.0:1789760770.626729:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.626750:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6434, 6434] 00000008:00010000:2.0:1789760770.626772:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.626779:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.626792:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26353664, 26357760) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.626806:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26353664=0x1922000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.626876:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.626879:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26357760 00000020:00010000:2.0:1789760770.626888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.626892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.626895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.626900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.626904:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.626913:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.626919:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.626946:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760770.627062:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26353664<>26357760 00000100:00100000:3.0:1789760770.627602:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.683236:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973753 gen 3 00000008:00010000:3.0:1789760770.683287:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760770.683295:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.683515:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.683611:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760770.683653:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808128:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.687285:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6434, 6434] 00000080:00200000:2.0:1789760770.687382:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.687390:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.687531:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26357760 i_blocks 53255 00000020:00200000:2.0:1789760770.687538:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.687556:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.687591:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26357760, count: 4096 00000080:00200000:2.0:1789760770.687600:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26357760, bytes: 4096 00000080:00200000:2.0:1789760770.687615:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.687625:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.687627:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26357760, 26361856) 0 0 00000080:00200000:2.0:1789760770.687632:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.687648:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6435, 6435] 00020000:00200000:2.0:1789760770.687658:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26357760 chunk: [26357760, 26361856) 26361856, 4096 00020000:00200000:2.0:1789760770.687663:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.687684:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6435, 6435] 00000008:00010000:2.0:1789760770.687703:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.687709:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.687722:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26357760, 26361856) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.687737:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26357760=0x1923000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.687803:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.687806:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26361856 00000020:00010000:2.0:1789760770.687814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.687818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.687821:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.687825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.687829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.687839:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.687845:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.687871:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760770.687981:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26357760<>26361856 00000100:00100000:3.0:1789760770.688415:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760770.688444:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808256:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.751739:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973754 gen 3 00000008:00010000:3.0:1789760770.751850:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760770.751859:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.752010:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.752080:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760770.752124:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.752207:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6435, 6435] 00000080:00200000:2.0:1789760770.752225:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.752231:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.752247:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26361856 i_blocks 53255 00000020:00200000:2.0:1789760770.752252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.752264:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.752298:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26361856, count: 4096 00000080:00200000:2.0:1789760770.752306:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26361856, bytes: 4096 00000080:00200000:2.0:1789760770.752317:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.752327:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.752329:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26361856, 26365952) 0 0 00000080:00200000:2.0:1789760770.752334:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.752350:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6436, 6436] 00020000:00200000:2.0:1789760770.752359:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26361856 chunk: [26361856, 26365952) 26365952, 4096 00020000:00200000:2.0:1789760770.752363:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.752382:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6436, 6436] 00000008:00010000:2.0:1789760770.752421:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.752428:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.752439:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26361856, 26365952) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.752451:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26361856=0x1924000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.752505:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.752508:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26365952 00000020:00010000:2.0:1789760770.752516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.752519:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.752522:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.752526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.752529:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.752537:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.752544:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.752559:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760770.752740:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26361856<>26365952 00000100:00100000:3.0:1789760770.752809:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.797278:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973755 gen 3 00000008:00010000:3.0:1789760770.797329:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760770.797337:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.797348:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.797369:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760770.797875:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.798130:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6436, 6436] 00000080:00200000:2.0:1789760770.798158:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.798165:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.798180:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26365952 i_blocks 53255 00000020:00200000:2.0:1789760770.798185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.798207:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.799380:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26365952, count: 4096 00000080:00200000:2.0:1789760770.799425:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26365952, bytes: 4096 00000080:00200000:2.0:1789760770.799440:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.799449:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.799452:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26365952, 26370048) 0 0 00000080:00200000:2.0:1789760770.799457:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.799475:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6437, 6437] 00020000:00200000:2.0:1789760770.799484:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26365952 chunk: [26365952, 26370048) 26370048, 4096 00020000:00200000:2.0:1789760770.799626:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.799651:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6437, 6437] 00000008:00010000:2.0:1789760770.799670:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.799675:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.799687:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26365952, 26370048) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.799700:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26365952=0x1925000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.800058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.800062:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26370048 00000020:00010000:2.0:1789760770.800073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.800077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.800080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.800084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.800087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.800100:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.800107:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.800711:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760770.800845:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26365952<>26370048 00000100:00100000:3.0:1789760770.801544:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760770.801579:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.845042:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973756 gen 3 00000008:00010000:3.0:1789760770.845092:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760770.845101:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.845122:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.845186:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760770.845336:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.845443:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6437, 6437] 00000080:00200000:2.0:1789760770.845458:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.845464:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.845478:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26370048 i_blocks 53255 00000020:00200000:2.0:1789760770.845481:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.845492:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.845718:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26370048, count: 4096 00000080:00200000:2.0:1789760770.845726:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26370048, bytes: 4096 00000080:00200000:2.0:1789760770.845737:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.845745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.845747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26370048, 26374144) 0 0 00000080:00200000:2.0:1789760770.845752:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.845816:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6438, 6438] 00020000:00200000:2.0:1789760770.845824:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26370048 chunk: [26370048, 26374144) 26374144, 4096 00020000:00200000:2.0:1789760770.845829:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.845848:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6438, 6438] 00000008:00010000:2.0:1789760770.845865:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.845927:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.845939:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26370048, 26374144) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.845953:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26370048=0x1926000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.846005:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.846008:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26374144 00000020:00010000:2.0:1789760770.846016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.846019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.846022:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.846026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.846029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.846037:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.846042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.846058:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760770.846324:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26370048<>26374144 00000100:00100000:3.0:1789760770.846390:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.897880:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973757 gen 3 00000008:00010000:3.0:1789760770.897936:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760770.897946:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.897958:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.898017:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760770.898051:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.898138:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6438, 6438] 00000080:00200000:2.0:1789760770.898157:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.898162:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.898235:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26374144 i_blocks 53255 00000020:00200000:2.0:1789760770.898239:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.898252:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.898285:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26374144, count: 4096 00000080:00200000:2.0:1789760770.898293:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26374144, bytes: 4096 00000080:00200000:2.0:1789760770.898301:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.898310:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.898313:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26374144, 26378240) 0 0 00000080:00200000:2.0:1789760770.898318:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.898335:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6439, 6439] 00020000:00200000:2.0:1789760770.898344:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26374144 chunk: [26374144, 26378240) 26378240, 4096 00020000:00200000:2.0:1789760770.898349:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.898369:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6439, 6439] 00000008:00010000:2.0:1789760770.898388:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.898425:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.898439:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26374144, 26378240) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.898455:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26374144=0x1927000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.898585:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.898589:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26378240 00000020:00010000:2.0:1789760770.898668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.898672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.898675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.898679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.898683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.898691:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.898699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.898716:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760770.898832:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26374144<>26378240 00000100:00100000:3.0:1789760770.898906:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493808768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.932495:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973758 gen 3 00000008:00010000:3.0:1789760770.932550:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760770.932558:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.932570:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.932678:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6439, 6439] 00000080:00200000:2.0:1789760770.932698:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.932704:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.932720:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26378240 i_blocks 53255 00000020:00200000:2.0:1789760770.932723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.932740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.932775:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26378240, count: 4096 00000080:00200000:2.0:1789760770.932784:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26378240, bytes: 4096 00000080:00200000:2.0:1789760770.932794:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.932804:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.932806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26378240, 26382336) 0 0 00000080:00200000:2.0:1789760770.932811:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.932827:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6440, 6440] 00020000:00200000:2.0:1789760770.932836:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26378240 chunk: [26378240, 26382336) 26382336, 4096 00020000:00200000:2.0:1789760770.932841:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.932862:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6440, 6440] 00000008:00010000:2.0:1789760770.932880:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.932887:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.932898:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26378240, 26382336) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.932913:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26378240=0x1928000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.932977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.932980:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26382336 00000020:00010000:2.0:1789760770.932988:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.932991:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.932994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.932998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.933001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.933009:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.933016:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.933034:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760770.933147:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26378240<>26382336 00000100:00100000:2.0:1789760770.933208:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808896:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760770.934138:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760770.934525:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493808768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760770.980696:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973759 gen 3 00000008:00010000:3.0:1789760770.980740:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760770.980748:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.980759:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.980783:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760770.980809:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493808896:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760770.980954:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6440, 6440] 00000080:00200000:2.0:1789760770.980969:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.980975:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.980990:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26382336 i_blocks 53255 00000020:00200000:2.0:1789760770.980994:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.981005:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.981037:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26382336, count: 4096 00000080:00200000:2.0:1789760770.981044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26382336, bytes: 4096 00000080:00200000:2.0:1789760770.981052:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.981060:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.981063:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26382336, 26386432) 0 0 00000080:00200000:2.0:1789760770.981068:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.981082:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6441, 6441] 00020000:00200000:2.0:1789760770.981090:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26382336 chunk: [26382336, 26386432) 26386432, 4096 00020000:00200000:2.0:1789760770.981094:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.982557:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6441, 6441] 00000008:00010000:2.0:1789760770.982587:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.982594:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760770.982608:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26382336, 26386432) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.982624:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26382336=0x1929000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.982691:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.982695:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26386432 00000020:00010000:2.0:1789760770.982703:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.982706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.982710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.982714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.982717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.982727:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.982735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.982758:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760770.982860:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26382336<>26386432 00000100:00100000:3.0:1789760770.983683:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760770.983712:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809024:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.020519:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760771.020533:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.020542:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.020559:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000080:00200000:2.0:1789760771.020586:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6441, 6441] 00000100:00100000:3.0:1789760771.020597:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.020605:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.020611:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.020625:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26386432 i_blocks 53255 00000020:00200000:2.0:1789760771.020628:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.020639:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.020689:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26386432, count: 4096 00000080:00200000:2.0:1789760771.020697:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26386432, bytes: 4096 00000080:00200000:2.0:1789760771.020706:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.020716:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.020718:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26386432, 26390528) 0 0 00000080:00200000:2.0:1789760771.020723:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.020740:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6442, 6442] 00020000:00200000:2.0:1789760771.020749:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26386432 chunk: [26386432, 26390528) 26390528, 4096 00020000:00200000:2.0:1789760771.020753:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.020778:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6442, 6442] 00000008:00010000:2.0:1789760771.020797:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.020803:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.020815:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26386432, 26390528) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.020832:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26386432=0x192a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.020979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.020983:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26390528 00000020:00010000:2.0:1789760771.020992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.020995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.020999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.021003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.021006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.021014:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.021022:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.021040:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760771.021155:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26386432<>26390528 00000100:00100000:3.0:1789760771.021317:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.056159:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973761 gen 3 00000008:00010000:3.0:1789760771.056287:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760771.056296:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.056307:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.056363:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760771.056430:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.056571:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6442, 6442] 00000080:00200000:2.0:1789760771.056587:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.056658:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.056757:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26390528 i_blocks 53255 00000020:00200000:2.0:1789760771.056762:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.056773:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.056809:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26390528, count: 4096 00000080:00200000:2.0:1789760771.056876:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26390528, bytes: 4096 00000080:00200000:2.0:1789760771.056889:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.056898:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.056900:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26390528, 26394624) 0 0 00000080:00200000:2.0:1789760771.056905:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.056923:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6443, 6443] 00020000:00200000:2.0:1789760771.056931:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26390528 chunk: [26390528, 26394624) 26394624, 4096 00020000:00200000:2.0:1789760771.056936:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.056957:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6443, 6443] 00000008:00010000:2.0:1789760771.056976:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.056982:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.056993:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26390528, 26394624) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.057223:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26390528=0x192b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.057287:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.057290:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26394624 00000020:00010000:2.0:1789760771.057299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.057302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.057306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.057309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.057312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.057320:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.057327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.057343:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760771.057467:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26390528<>26394624 00000100:00100000:3.0:1789760771.057540:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493809280:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.108480:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760771.108495:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.108508:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.108586:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760771.108619:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493809280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.108751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6443, 6443] 00000080:00200000:2.0:1789760771.108767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.108773:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.108787:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26394624 i_blocks 53255 00000020:00200000:2.0:1789760771.108791:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.108802:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.108836:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26394624, count: 4096 00000080:00200000:2.0:1789760771.108844:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26394624, bytes: 4096 00000080:00200000:2.0:1789760771.108856:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.108865:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.108938:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26394624, 26398720) 0 0 00000080:00200000:2.0:1789760771.108945:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.108963:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6444, 6444] 00020000:00200000:2.0:1789760771.108971:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26394624 chunk: [26394624, 26398720) 26398720, 4096 00020000:00200000:2.0:1789760771.108976:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.108997:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6444, 6444] 00000008:00010000:2.0:1789760771.109088:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.109095:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.109107:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26394624, 26398720) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.109130:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26394624=0x192c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.109255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.109258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26398720 00000020:00010000:2.0:1789760771.109268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.109271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.109276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.109279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.109282:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.109290:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.109297:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.109314:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760771.109691:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26394624<>26398720 00000100:00100000:2.0:1789760771.109743:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.142766:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973763 gen 3 00000008:00010000:3.0:1789760771.142811:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760771.142819:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.142827:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.142849:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760771.142875:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809408:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760771.142975:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6444, 6444] 00000080:00200000:0.0:1789760771.142989:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.142994:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.143019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26398720 i_blocks 53255 00000020:00200000:0.0:1789760771.143023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.143032:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.143060:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26398720, count: 4096 00000080:00200000:0.0:1789760771.143066:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26398720, bytes: 4096 00000080:00200000:0.0:1789760771.143074:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.143081:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.143083:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26398720, 26402816) 0 0 00000080:00200000:0.0:1789760771.143088:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.143100:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6445, 6445] 00020000:00200000:0.0:1789760771.143106:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26398720 chunk: [26398720, 26402816) 26402816, 4096 00020000:00200000:0.0:1789760771.143120:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.143135:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6445, 6445] 00000008:00010000:0.0:1789760771.143148:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.143152:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760771.143161:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26398720, 26402816) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.143170:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26398720=0x192d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.143214:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.143217:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26402816 00000020:00010000:0.0:1789760771.143223:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.143226:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.143229:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.143232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.143234:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.143240:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.143246:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.143259:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760771.143344:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 26398720<>26402816 00000100:00100000:1.0:1789760771.143565:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.143580:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493809536:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760771.208959:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760771.208975:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.208986:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.209007:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760771.209036:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493809536:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760771.209297:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6445, 6445] 00000080:00200000:0.0:1789760771.209317:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.209323:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.209335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26402816 i_blocks 53255 00000020:00200000:0.0:1789760771.209340:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.209352:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.209376:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26402816, count: 4096 00000080:00200000:0.0:1789760771.209383:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26402816, bytes: 4096 00000080:00200000:0.0:1789760771.209433:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.209441:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.209443:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26402816, 26406912) 0 0 00000080:00200000:0.0:1789760771.209448:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.209460:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6446, 6446] 00020000:00200000:0.0:1789760771.209467:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26402816 chunk: [26402816, 26406912) 26406912, 4096 00020000:00200000:0.0:1789760771.209471:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.209487:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6446, 6446] 00000008:00010000:0.0:1789760771.209502:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.209506:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760771.209516:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26402816, 26406912) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.209527:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26402816=0x192e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.209576:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.209579:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26406912 00000020:00010000:0.0:1789760771.209586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.209589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.209592:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.209595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.209597:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.209603:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.209609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.209626:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760771.209706:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 26402816<>26406912 00000100:00100000:1.0:1789760771.209804:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493809664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760771.267986:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973765 gen 3 00000008:00010000:1.0:1789760771.268053:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760771.268062:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.268176:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.268259:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760771.268288:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493809664:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760771.268373:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6446, 6446] 00000080:00200000:0.0:1789760771.268486:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.268493:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.268509:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26406912 i_blocks 53255 00000020:00200000:0.0:1789760771.268513:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.268524:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.268561:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26406912, count: 4096 00000080:00200000:0.0:1789760771.268569:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26406912, bytes: 4096 00000080:00200000:0.0:1789760771.268578:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.268621:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.268624:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26406912, 26411008) 0 0 00000080:00200000:0.0:1789760771.268631:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.268645:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6447, 6447] 00020000:00200000:0.0:1789760771.268654:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26406912 chunk: [26406912, 26411008) 26411008, 4096 00020000:00200000:0.0:1789760771.268658:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.268678:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6447, 6447] 00000008:00010000:0.0:1789760771.268692:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.268697:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760771.268707:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26406912, 26411008) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.268719:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26406912=0x192f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.268778:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.268780:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26411008 00000020:00010000:0.0:1789760771.268786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.268789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.268791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.268793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.268796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.268803:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.268810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.268833:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760771.269958:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 26406912<>26411008 00000100:00100000:1.0:1789760771.270024:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493809792:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760771.311296:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973766 gen 3 00000008:00010000:1.0:1789760771.311348:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760771.311357:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.311367:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1789760771.311467:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760771.311524:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760771.311552:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493809792:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760771.311697:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6447, 6447] 00000080:00200000:3.0:1789760771.311714:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.311721:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.311739:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26411008 i_blocks 53255 00000020:00200000:3.0:1789760771.311743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.311754:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.311791:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26411008, count: 4096 00000080:00200000:3.0:1789760771.311798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26411008, bytes: 4096 00000080:00200000:3.0:1789760771.311809:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.311819:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.311821:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26411008, 26415104) 0 0 00000080:00200000:3.0:1789760771.311826:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.311841:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6448, 6448] 00020000:00200000:3.0:1789760771.311849:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26411008 chunk: [26411008, 26415104) 26415104, 4096 00020000:00200000:3.0:1789760771.311853:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.311874:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6448, 6448] 00000008:00010000:3.0:1789760771.311893:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.311899:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760771.311911:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26411008, 26415104) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.311927:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26411008=0x1930000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.311983:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.311986:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26415104 00000020:00010000:3.0:1789760771.311994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.311997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.312000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.312004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.312007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.312014:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.312021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.312037:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760771.312795:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 26411008<>26415104 00000100:00100000:2.0:1789760771.312856:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760771.312942:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809920:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.350912:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760771.350926:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.350936:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.350955:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760771.350982:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493809920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.351133:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6448, 6448] 00000080:00200000:2.0:1789760771.351150:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.351157:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.351171:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26415104 i_blocks 53255 00000020:00200000:2.0:1789760771.351175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.351186:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.351232:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26415104, count: 4096 00000080:00200000:2.0:1789760771.351240:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26415104, bytes: 4096 00000080:00200000:2.0:1789760771.351251:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.351260:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.351262:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26415104, 26419200) 0 0 00000080:00200000:2.0:1789760771.351266:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.351283:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6449, 6449] 00020000:00200000:2.0:1789760771.351293:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26415104 chunk: [26415104, 26419200) 26419200, 4096 00020000:00200000:2.0:1789760771.351297:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.351329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6449, 6449] 00000008:00010000:2.0:1789760771.351348:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.351354:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.351365:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26415104, 26419200) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.351380:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26415104=0x1931000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.351470:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.351473:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26419200 00000020:00010000:2.0:1789760771.351481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.351484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.351487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.351491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.351494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.351501:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.351508:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.351525:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760771.351647:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26415104<>26419200 00000100:00100000:3.0:1789760771.352187:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.388477:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973767 gen 3 00000100:00100000:3.0:1789760771.388521:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83800 x1876699493810048/t4294973768(4294973768) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760787 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760771.388573:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760771.388580:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.388604:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.388674:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760771.388702:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810048:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.391764:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6449, 6449] 00000080:00200000:2.0:1789760771.393048:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.393055:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.393073:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26419200 i_blocks 53255 00000020:00200000:2.0:1789760771.393079:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.393098:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.393141:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26419200, count: 4096 00000080:00200000:2.0:1789760771.393151:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26419200, bytes: 4096 00000080:00200000:2.0:1789760771.393166:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.393176:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.393179:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26419200, 26423296) 0 0 00000080:00200000:2.0:1789760771.393183:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.393198:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6450, 6450] 00020000:00200000:2.0:1789760771.393206:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26419200 chunk: [26419200, 26423296) 26423296, 4096 00020000:00200000:2.0:1789760771.393210:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.393232:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6450, 6450] 00000008:00010000:2.0:1789760771.393254:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.393260:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.393273:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26419200, 26423296) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.393288:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26419200=0x1932000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.393359:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.393363:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26423296 00000020:00010000:2.0:1789760771.393371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.393375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.393378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.393381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.393384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.393509:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.393518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.393546:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760771.393996:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26419200<>26423296 00000100:00100000:3.0:1789760771.394084:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493810176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.449267:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973769 gen 3 00000008:00010000:3.0:1789760771.449338:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760771.449348:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.449359:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.449383:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760771.449443:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493810176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.449540:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6450, 6450] 00000080:00200000:2.0:1789760771.449557:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.449563:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.449580:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26423296 i_blocks 53255 00000020:00200000:2.0:1789760771.449583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.449747:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.449928:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26423296, count: 4096 00000080:00200000:2.0:1789760771.449937:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26423296, bytes: 4096 00000080:00200000:2.0:1789760771.449950:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.449959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.449961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26423296, 26427392) 0 0 00000080:00200000:2.0:1789760771.449966:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.449983:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6451, 6451] 00020000:00200000:2.0:1789760771.449991:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26423296 chunk: [26423296, 26427392) 26427392, 4096 00020000:00200000:2.0:1789760771.449996:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.450016:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6451, 6451] 00000008:00010000:2.0:1789760771.450036:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.450042:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.450056:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26423296, 26427392) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.450070:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26423296=0x1933000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.450142:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.450145:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26427392 00000020:00010000:2.0:1789760771.450154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.450157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.450160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.450164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.450830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.450844:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.450852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.450875:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760771.450997:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26423296<>26427392 00000100:00100000:3.0:1789760771.451505:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.484482:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973770 gen 3 00000008:00010000:3.0:1789760771.484536:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760771.484545:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.484556:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.484577:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760771.484614:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810304:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.484726:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6451, 6451] 00000080:00200000:2.0:1789760771.484744:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.484750:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.484766:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26427392 i_blocks 53255 00000020:00200000:2.0:1789760771.484784:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.484796:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.484831:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26427392, count: 4096 00000080:00200000:2.0:1789760771.484839:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26427392, bytes: 4096 00000080:00200000:2.0:1789760771.484848:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.484856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.484859:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26427392, 26431488) 0 0 00000080:00200000:2.0:1789760771.484864:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.484880:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6452, 6452] 00020000:00200000:2.0:1789760771.484889:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26427392 chunk: [26427392, 26431488) 26431488, 4096 00020000:00200000:2.0:1789760771.484894:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.484913:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6452, 6452] 00000008:00010000:2.0:1789760771.484932:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.484938:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.484949:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26427392, 26431488) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.484962:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26427392=0x1934000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.485024:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.485027:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26431488 00000020:00010000:2.0:1789760771.485036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.485040:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.485050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.485054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.485057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.485065:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.485072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.485088:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760771.485213:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26427392<>26431488 00000100:00100000:3.0:1789760771.485413:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760771.485432:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.524758:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973771 gen 3 00000008:00010000:3.0:1789760771.524809:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760771.524818:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.524830:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.524904:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760771.524935:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.525099:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6452, 6452] 00000080:00200000:2.0:1789760771.525125:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.525132:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.525148:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26431488 i_blocks 53255 00000020:00200000:2.0:1789760771.525152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.525163:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.525248:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26431488, count: 4096 00000080:00200000:2.0:1789760771.525257:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26431488, bytes: 4096 00000080:00200000:2.0:1789760771.525267:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.525276:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.525278:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26431488, 26435584) 0 0 00000080:00200000:2.0:1789760771.525283:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.525299:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6453, 6453] 00020000:00200000:2.0:1789760771.525308:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26431488 chunk: [26431488, 26435584) 26435584, 4096 00020000:00200000:2.0:1789760771.525312:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.525329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6453, 6453] 00000008:00010000:2.0:1789760771.525346:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.525353:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.525364:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26431488, 26435584) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.525378:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26431488=0x1935000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.525545:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.525549:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26435584 00000020:00010000:2.0:1789760771.525673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.525677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.525680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.525683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.525685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.525693:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.525699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.525713:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760771.525809:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26431488<>26435584 00000100:00100000:3.0:1789760771.525874:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810560:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.559361:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973772 gen 3 00000008:00010000:3.0:1789760771.559430:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760771.559439:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.559451:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.559469:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760771.559499:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810560:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.559598:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6453, 6453] 00000080:00200000:2.0:1789760771.559615:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.559621:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.559638:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26435584 i_blocks 53255 00000020:00200000:2.0:1789760771.559642:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.559654:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.559688:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26435584, count: 4096 00000080:00200000:2.0:1789760771.559696:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26435584, bytes: 4096 00000080:00200000:2.0:1789760771.559705:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.559714:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.559716:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26435584, 26439680) 0 0 00000080:00200000:2.0:1789760771.559721:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.559739:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6454, 6454] 00020000:00200000:2.0:1789760771.559747:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26435584 chunk: [26435584, 26439680) 26439680, 4096 00020000:00200000:2.0:1789760771.559752:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.559771:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6454, 6454] 00000008:00010000:2.0:1789760771.559789:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.559795:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.559807:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26435584, 26439680) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.559822:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26435584=0x1936000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.559881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.559884:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26439680 00000020:00010000:2.0:1789760771.559895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.559898:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.559901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.559905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.559907:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.559916:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.559923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.559939:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760771.560044:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26435584<>26439680 00000100:00100000:3.0:1789760771.560211:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760771.560230:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810688:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760771.616120:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760771.616137:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760771.616150:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760771.616174:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760771.616204:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810688:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760771.616302:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6454, 6454] 00000080:00200000:1.0:1789760771.616318:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.616324:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.616339:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26439680 i_blocks 53255 00000020:00200000:1.0:1789760771.616343:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.616354:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.616384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26439680, count: 4096 00000080:00200000:1.0:1789760771.616391:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26439680, bytes: 4096 00000080:00200000:1.0:1789760771.616420:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.616429:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.616431:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26439680, 26443776) 0 0 00000080:00200000:1.0:1789760771.616436:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.616452:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6455, 6455] 00020000:00200000:1.0:1789760771.616471:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26439680 chunk: [26439680, 26443776) 26443776, 4096 00020000:00200000:1.0:1789760771.616476:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.616495:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6455, 6455] 00000008:00010000:1.0:1789760771.616512:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.616518:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760771.616529:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26439680, 26443776) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.616542:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26439680=0x1937000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.616678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.616682:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26443776 00000020:00010000:1.0:1789760771.616691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.616695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.616699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.616702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.616705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.616715:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.616722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.616736:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760771.616832:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 26439680<>26443776 00000100:00100000:1.0:1789760771.616872:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760771.616888:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493810816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760771.658579:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973773 gen 3 00000100:00100000:1.0:1789760771.658623:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016f480 x1876699493810816/t4294973774(4294973774) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760787 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760771.658673:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:1.0:1789760771.658680:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.658691:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.658715:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:1.0:1789760771.658739:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493810816:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.658954:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6455, 6455] 00000080:00200000:2.0:1789760771.658971:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.658976:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.658990:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26443776 i_blocks 53255 00000020:00200000:2.0:1789760771.658994:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.659003:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.659035:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26443776, count: 4096 00000080:00200000:2.0:1789760771.659043:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26443776, bytes: 4096 00000080:00200000:2.0:1789760771.659066:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.659075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.659077:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26443776, 26447872) 0 0 00000080:00200000:2.0:1789760771.659081:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.659096:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6456, 6456] 00020000:00200000:2.0:1789760771.659115:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26443776 chunk: [26443776, 26447872) 26447872, 4096 00020000:00200000:2.0:1789760771.659120:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.659139:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6456, 6456] 00000008:00010000:2.0:1789760771.659156:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.659373:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.659388:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26443776, 26447872) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.659424:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26443776=0x1938000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.659484:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.659486:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26447872 00000020:00010000:2.0:1789760771.659495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.659499:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.659502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.659506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.659508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.659516:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.659523:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.659537:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760771.659647:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26443776<>26447872 00000100:00100000:2.0:1789760771.659717:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.692369:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973775 gen 3 00000008:00010000:3.0:1789760771.692483:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760771.692493:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.692503:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.692564:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760771.692592:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493810944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.692666:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6456, 6456] 00000080:00200000:2.0:1789760771.692683:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.692690:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.692705:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26447872 i_blocks 53255 00000020:00200000:2.0:1789760771.692709:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.692718:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.692751:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26447872, count: 4096 00000080:00200000:2.0:1789760771.692758:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26447872, bytes: 4096 00000080:00200000:2.0:1789760771.692769:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.692778:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.692781:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26447872, 26451968) 0 0 00000080:00200000:2.0:1789760771.692786:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.692801:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6457, 6457] 00020000:00200000:2.0:1789760771.692809:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26447872 chunk: [26447872, 26451968) 26451968, 4096 00020000:00200000:2.0:1789760771.692814:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.692832:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6457, 6457] 00000008:00010000:2.0:1789760771.692915:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.692921:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.692932:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26447872, 26451968) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.692946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26447872=0x1939000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.693065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.693068:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26451968 00000020:00010000:2.0:1789760771.693077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.693082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.693085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.693089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.693093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.693100:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.693117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.693132:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760771.694062:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26447872<>26451968 00000100:00100000:3.0:1789760771.694149:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811072:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.758805:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760771.758822:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.759041:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.759100:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760771.759141:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.759211:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6457, 6457] 00000080:00200000:2.0:1789760771.759227:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.759233:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.759248:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26451968 i_blocks 53255 00000020:00200000:2.0:1789760771.759252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.759262:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.759294:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26451968, count: 4096 00000080:00200000:2.0:1789760771.759300:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26451968, bytes: 4096 00000080:00200000:2.0:1789760771.759309:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.759318:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.759320:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26451968, 26456064) 0 0 00000080:00200000:2.0:1789760771.759391:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.759427:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6458, 6458] 00020000:00200000:2.0:1789760771.759436:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26451968 chunk: [26451968, 26456064) 26456064, 4096 00020000:00200000:2.0:1789760771.759440:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.759462:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6458, 6458] 00000008:00010000:2.0:1789760771.759480:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.759486:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.759498:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26451968, 26456064) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.759511:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26451968=0x193a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.759630:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.759633:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26456064 00000020:00010000:2.0:1789760771.759643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.759647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.759650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.759653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.759656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.759665:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.759672:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.759687:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760771.759793:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26451968<>26456064 00000100:00100000:3.0:1789760771.759879:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493811200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760771.841060:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973777 gen 3 00000008:00010000:3.0:1789760771.841135:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760771.841146:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.841166:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.841184:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760771.841212:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493811200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760771.841428:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6458, 6458] 00000080:00200000:2.0:1789760771.841445:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760771.841452:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760771.841468:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26456064 i_blocks 53255 00000020:00200000:2.0:1789760771.841472:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760771.841483:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760771.841516:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26456064, count: 4096 00000080:00200000:2.0:1789760771.841523:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26456064, bytes: 4096 00000080:00200000:2.0:1789760771.841533:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760771.841541:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760771.841544:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26456064, 26460160) 0 0 00000080:00200000:2.0:1789760771.841549:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760771.841565:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6459, 6459] 00020000:00200000:2.0:1789760771.841574:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26456064 chunk: [26456064, 26460160) 26460160, 4096 00020000:00200000:2.0:1789760771.841578:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760771.841765:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6459, 6459] 00000008:00010000:2.0:1789760771.841786:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760771.841793:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760771.841807:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26456064, 26460160) append 0 append lockpos 0 00000080:00200000:2.0:1789760771.841937:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26456064=0x193b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760771.842002:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760771.842005:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26460160 00000020:00010000:2.0:1789760771.842014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760771.842018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.842021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760771.842025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760771.842028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760771.842036:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760771.842043:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760771.842071:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760771.842189:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26456064<>26460160 00000100:00100000:3.0:1789760771.842265:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811328:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.889544:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760771.889560:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.889572:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.889697:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760771.889726:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811328:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760771.889942:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6459, 6459] 00000080:00200000:0.0:1789760771.889960:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.889967:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.889985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26460160 i_blocks 53255 00000020:00200000:0.0:1789760771.889989:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.890001:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.890120:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26460160, count: 4096 00000080:00200000:0.0:1789760771.890130:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26460160, bytes: 4096 00000080:00200000:0.0:1789760771.890259:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.890269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.890272:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26460160, 26464256) 0 0 00000080:00200000:0.0:1789760771.890277:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.890295:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6460, 6460] 00020000:00200000:0.0:1789760771.890305:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26460160 chunk: [26460160, 26464256) 26464256, 4096 00020000:00200000:0.0:1789760771.890311:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.890331:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6460, 6460] 00000008:00010000:0.0:1789760771.890351:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.890358:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760771.890370:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26460160, 26464256) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.890385:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26460160=0x193c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.890477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.890481:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26464256 00000020:00010000:0.0:1789760771.890489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.890493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.890495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.890498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.890506:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.890514:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.890520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.890538:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760771.890723:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 26460160<>26464256 00000100:00100000:1.0:1789760771.890829:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811456:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760771.923516:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973779 gen 3 00000008:00010000:1.0:1789760771.923635:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760771.923644:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.923655:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.923675:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760771.923703:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811456:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760771.924432:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6460, 6460] 00000080:00200000:3.0:1789760771.924450:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.924457:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.924472:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26464256 i_blocks 53255 00000020:00200000:3.0:1789760771.924476:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.924487:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.924519:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26464256, count: 4096 00000080:00200000:3.0:1789760771.924526:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26464256, bytes: 4096 00000080:00200000:3.0:1789760771.924537:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.924546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.924548:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26464256, 26468352) 0 0 00000080:00200000:3.0:1789760771.924553:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.925806:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6461, 6461] 00020000:00200000:3.0:1789760771.925821:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26464256 chunk: [26464256, 26468352) 26468352, 4096 00020000:00200000:3.0:1789760771.925826:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.925851:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6461, 6461] 00000008:00010000:3.0:1789760771.925872:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.925878:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760771.925891:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26464256, 26468352) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.925906:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26464256=0x193d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.925973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.925976:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26468352 00000020:00010000:3.0:1789760771.925984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.925987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.925989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.925993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.925996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.926007:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.926014:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.926048:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760771.926804:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 26464256<>26468352 00000100:00100000:2.0:1789760771.927824:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493811584:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760771.956520:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760771.956537:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.956548:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.956570:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000080:00200000:1.0:1789760771.956628:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6461, 6461] 00000080:00200000:1.0:1789760771.956647:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.956653:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760771.956662:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493811584:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760771.956669:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26468352 i_blocks 53255 00000020:00200000:1.0:1789760771.956673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.956683:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.956716:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26468352, count: 4096 00000080:00200000:1.0:1789760771.956723:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26468352, bytes: 4096 00000080:00200000:1.0:1789760771.956733:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.956751:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.956753:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26468352, 26472448) 0 0 00000080:00200000:1.0:1789760771.956759:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.956776:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6462, 6462] 00020000:00200000:1.0:1789760771.956784:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26468352 chunk: [26468352, 26472448) 26472448, 4096 00020000:00200000:1.0:1789760771.956793:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.956813:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6462, 6462] 00000008:00010000:1.0:1789760771.956840:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.956847:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760771.956860:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26468352, 26472448) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.956874:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26468352=0x193e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.956930:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.956938:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26472448 00000020:00010000:1.0:1789760771.956946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.956950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.956953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.956957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.956960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.956970:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.956978:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.956991:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760771.957092:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 26468352<>26472448 00000100:00100000:1.0:1789760771.957145:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760771.957222:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811712:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760771.994693:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973781 gen 3 00000008:00010000:1.0:1789760771.994756:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:1.0:1789760771.994764:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.994774:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.994796:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:1.0:1789760771.994835:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811712:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760771.994937:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6462, 6462] 00000080:00200000:0.0:1789760771.994953:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.994960:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.994975:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26472448 i_blocks 53255 00000020:00200000:0.0:1789760771.994979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.994989:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.995024:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26472448, count: 4096 00000080:00200000:0.0:1789760771.995031:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26472448, bytes: 4096 00000080:00200000:0.0:1789760771.995041:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.995051:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.995054:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26472448, 26476544) 0 0 00000080:00200000:0.0:1789760771.995058:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.995074:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6463, 6463] 00020000:00200000:0.0:1789760771.995082:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26472448 chunk: [26472448, 26476544) 26476544, 4096 00020000:00200000:0.0:1789760771.995321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.995351:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6463, 6463] 00000008:00010000:0.0:1789760771.995372:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.995527:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760771.995541:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26472448, 26476544) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.995555:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26472448=0x193f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.995615:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.995618:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26476544 00000020:00010000:0.0:1789760771.995625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.995629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.995632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.995635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.995638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.995646:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.995653:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.995672:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760771.995774:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 26472448<>26476544 00000100:00100000:1.0:1789760771.996111:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811840:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760772.048232:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973782 gen 3 00000008:00010000:1.0:1789760772.048282:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760772.048291:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.048303:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.048326:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760772.048355:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493811840:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760772.048575:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6463, 6463] 00000080:00200000:3.0:1789760772.048591:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.048598:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.048613:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26476544 i_blocks 53255 00000020:00200000:3.0:1789760772.048617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.048627:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.048660:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26476544, count: 4096 00000080:00200000:3.0:1789760772.048668:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26476544, bytes: 4096 00000080:00200000:3.0:1789760772.048678:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.048687:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.048689:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26476544, 26480640) 0 0 00000080:00200000:3.0:1789760772.048693:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.048711:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6464, 6464] 00020000:00200000:3.0:1789760772.048720:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26476544 chunk: [26476544, 26480640) 26480640, 4096 00020000:00200000:3.0:1789760772.048726:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.048746:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6464, 6464] 00000008:00010000:3.0:1789760772.048764:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.048771:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760772.048852:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26476544, 26480640) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.048869:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26476544=0x1940000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.048931:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.048934:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26480640 00000020:00010000:3.0:1789760772.048943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.048946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.048949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.048953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.048956:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.048967:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.048973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.048991:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760772.049094:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 26476544<>26480640 00000100:00100000:2.0:1789760772.049179:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811968:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760772.086885:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760772.087061:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.087072:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.087084:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760772.087126:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493811968:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.087342:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6464, 6464] 00000080:00200000:0.0:1789760772.087359:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.087366:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.087482:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26480640 i_blocks 53255 00000020:00200000:0.0:1789760772.087487:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.087534:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.087568:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26480640, count: 4096 00000080:00200000:0.0:1789760772.087575:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26480640, bytes: 4096 00000080:00200000:0.0:1789760772.087597:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.087605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.087609:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26480640, 26484736) 0 0 00000080:00200000:0.0:1789760772.087613:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.087629:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6465, 6465] 00020000:00200000:0.0:1789760772.087637:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26480640 chunk: [26480640, 26484736) 26484736, 4096 00020000:00200000:0.0:1789760772.087642:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.087664:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6465, 6465] 00000008:00010000:0.0:1789760772.087682:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.087689:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.087700:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26480640, 26484736) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.087713:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26480640=0x1941000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.087770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.087773:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26484736 00000020:00010000:0.0:1789760772.087781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.087784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.087787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.087790:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.087793:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.087800:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.087807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.088313:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760772.088550:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26480640<>26484736 00000100:00100000:0.0:1789760772.088603:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812096:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.133795:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973783 gen 3 00000100:00100000:0.0:1789760772.133835:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304f800 x1876699493812096/t4294973784(4294973784) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760788 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760772.133887:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760772.133894:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.133904:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.133932:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760772.133958:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812096:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760772.134137:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6465, 6465] 00000080:00200000:1.0:1789760772.134222:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.134230:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.134246:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26484736 i_blocks 53255 00000020:00200000:1.0:1789760772.134251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.134262:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.134298:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26484736, count: 4096 00000080:00200000:1.0:1789760772.134306:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26484736, bytes: 4096 00000080:00200000:1.0:1789760772.134321:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.134331:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.134334:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26484736, 26488832) 0 0 00000080:00200000:1.0:1789760772.134339:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.134356:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6466, 6466] 00020000:00200000:1.0:1789760772.134371:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26484736 chunk: [26484736, 26488832) 26488832, 4096 00020000:00200000:1.0:1789760772.134385:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.134465:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6466, 6466] 00000008:00010000:1.0:1789760772.134485:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.134492:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760772.134505:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26484736, 26488832) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.134520:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26484736=0x1942000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.134590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.134594:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26488832 00000020:00010000:1.0:1789760772.134602:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.134605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.134609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.134612:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.134615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.134626:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.134633:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.134646:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760772.134756:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 26484736<>26488832 00000100:00100000:0.0:1789760772.134911:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760772.134932:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812224:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.198727:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973785 gen 3 00000008:00010000:0.0:1789760772.198795:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760772.198803:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.198814:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.198845:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760772.198873:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812224:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760772.198932:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6466, 6466] 00000080:00200000:1.0:1789760772.198949:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.198954:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.198969:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26488832 i_blocks 53255 00000020:00200000:1.0:1789760772.198973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.198983:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.199017:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26488832, count: 4096 00000080:00200000:1.0:1789760772.199024:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26488832, bytes: 4096 00000080:00200000:1.0:1789760772.199034:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.199043:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.199046:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26488832, 26492928) 0 0 00000080:00200000:1.0:1789760772.199050:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.199067:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6467, 6467] 00020000:00200000:1.0:1789760772.199075:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26488832 chunk: [26488832, 26492928) 26492928, 4096 00020000:00200000:1.0:1789760772.199079:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.199111:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6467, 6467] 00000008:00010000:1.0:1789760772.199131:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.199138:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760772.199149:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26488832, 26492928) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.199164:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26488832=0x1943000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.199222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.199225:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26492928 00000020:00010000:1.0:1789760772.199233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.199237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.199240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.199244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.199252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.199264:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.199365:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.199384:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760772.199533:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 26488832<>26492928 00000100:00100000:0.0:1789760772.199611:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812352:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.248848:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973786 gen 3 00000008:00010000:0.0:1789760772.248901:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760772.248909:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.248920:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.248943:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760772.248975:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812352:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760772.249113:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6467, 6467] 00000080:00200000:3.0:1789760772.249130:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.249136:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.249152:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26492928 i_blocks 53255 00000020:00200000:3.0:1789760772.249298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.249310:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.249346:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26492928, count: 4096 00000080:00200000:3.0:1789760772.249353:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26492928, bytes: 4096 00000080:00200000:3.0:1789760772.249546:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.249556:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.249558:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26492928, 26497024) 0 0 00000080:00200000:3.0:1789760772.249563:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.249580:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6468, 6468] 00020000:00200000:3.0:1789760772.249589:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26492928 chunk: [26492928, 26497024) 26497024, 4096 00020000:00200000:3.0:1789760772.249593:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.249614:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6468, 6468] 00000008:00010000:3.0:1789760772.249633:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.249640:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760772.249651:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26492928, 26497024) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.249665:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26492928=0x1944000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.249724:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.249727:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26497024 00000020:00010000:3.0:1789760772.249735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.249739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.249748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.249752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.249755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.249764:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.249771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.249787:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760772.249889:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 26492928<>26497024 00000100:00100000:3.0:1789760772.249939:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.249960:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493812480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760772.306653:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973787 gen 3 00000008:00010000:3.0:1789760772.306699:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760772.306706:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.306715:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.306730:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000080:00200000:2.0:1789760772.306756:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6468, 6468] 00000100:00100000:3.0:1789760772.306757:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493812480:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.306772:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.306778:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.306794:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26497024 i_blocks 53255 00000020:00200000:2.0:1789760772.306797:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.306807:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.307008:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26497024, count: 4096 00000080:00200000:2.0:1789760772.307017:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26497024, bytes: 4096 00000080:00200000:2.0:1789760772.307028:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.307037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.307040:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26497024, 26501120) 0 0 00000080:00200000:2.0:1789760772.307045:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.307062:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6469, 6469] 00020000:00200000:2.0:1789760772.307075:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26497024 chunk: [26497024, 26501120) 26501120, 4096 00020000:00200000:2.0:1789760772.307080:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.307110:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6469, 6469] 00000008:00010000:2.0:1789760772.307130:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.307136:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.307147:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26497024, 26501120) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.307162:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26497024=0x1945000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.307289:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.307292:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26501120 00000020:00010000:2.0:1789760772.307301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.307305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.307308:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.307312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.307315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.307323:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.307330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.307347:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760772.307602:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26497024<>26501120 00000100:00100000:3.0:1789760772.307662:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493812608:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760772.334941:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760772.350362:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973788 gen 3 00000008:00010000:3.0:1789760772.350443:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760772.350452:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.350463:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.350488:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000080:00200000:0.0:1789760772.350522:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6469, 6469] 00000100:00100000:3.0:1789760772.350527:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493812608:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.350539:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.350543:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.350558:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26501120 i_blocks 53255 00000020:00200000:0.0:1789760772.350562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.350573:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.350607:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26501120, count: 4096 00000080:00200000:0.0:1789760772.350615:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26501120, bytes: 4096 00000080:00200000:0.0:1789760772.350626:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.350635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.350638:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26501120, 26505216) 0 0 00000080:00200000:0.0:1789760772.350643:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.350660:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6470, 6470] 00020000:00200000:0.0:1789760772.350667:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26501120 chunk: [26501120, 26505216) 26505216, 4096 00020000:00200000:0.0:1789760772.350672:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.350693:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6470, 6470] 00000008:00010000:0.0:1789760772.350712:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.350718:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.350730:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26501120, 26505216) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.350743:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26501120=0x1946000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.350803:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.350805:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26505216 00000020:00010000:0.0:1789760772.350814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.350831:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.350835:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.350839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.350841:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.350849:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.350856:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.350872:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760772.350974:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26501120<>26505216 00000100:00100000:1.0:1789760772.351067:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493812736:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760772.388545:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973789 gen 3 00000008:00010000:1.0:1789760772.388609:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760772.388617:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.388628:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.388651:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760772.388679:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493812736:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.388787:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6470, 6470] 00000080:00200000:0.0:1789760772.388802:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.388809:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.388824:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26505216 i_blocks 53255 00000020:00200000:0.0:1789760772.388827:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.388838:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.388870:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26505216, count: 4096 00000080:00200000:0.0:1789760772.388877:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26505216, bytes: 4096 00000080:00200000:0.0:1789760772.388887:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.388896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.388898:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26505216, 26509312) 0 0 00000080:00200000:0.0:1789760772.388903:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.388920:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6471, 6471] 00020000:00200000:0.0:1789760772.388928:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26505216 chunk: [26505216, 26509312) 26509312, 4096 00020000:00200000:0.0:1789760772.388932:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.388952:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6471, 6471] 00000008:00010000:0.0:1789760772.388969:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.388975:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.388988:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26505216, 26509312) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.389003:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26505216=0x1947000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.389062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.389065:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26509312 00000020:00010000:0.0:1789760772.389073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.389077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.389080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.389083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.389087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.389095:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.389112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.389128:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760772.389297:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 26505216<>26509312 00000100:00100000:1.0:1789760772.389688:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812864:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760772.421426:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973790 gen 3 00000008:00010000:1.0:1789760772.421477:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760772.421486:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.421496:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.421517:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760772.421544:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812864:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.422019:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6471, 6471] 00000080:00200000:0.0:1789760772.422037:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.422043:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.422060:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26509312 i_blocks 53255 00000020:00200000:0.0:1789760772.422065:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.422075:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.422120:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26509312, count: 4096 00000080:00200000:0.0:1789760772.422128:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26509312, bytes: 4096 00000080:00200000:0.0:1789760772.422137:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.422146:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.422150:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26509312, 26513408) 0 0 00000080:00200000:0.0:1789760772.422162:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.422179:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6472, 6472] 00020000:00200000:0.0:1789760772.422188:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26509312 chunk: [26509312, 26513408) 26513408, 4096 00020000:00200000:0.0:1789760772.422192:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.422211:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6472, 6472] 00000008:00010000:0.0:1789760772.422229:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.422235:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.422247:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26509312, 26513408) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.422261:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26509312=0x1948000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.422317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.422320:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26513408 00000020:00010000:0.0:1789760772.422330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.422333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.422336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.422340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.422342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.422350:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.422357:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.422373:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760772.422510:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26509312<>26513408 00000100:00100000:1.0:1789760772.422742:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760772.422765:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812992:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760772.455444:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760772.455460:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.455472:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.455496:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760772.455526:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493812992:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.455631:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6472, 6472] 00000080:00200000:0.0:1789760772.455648:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.455654:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.455669:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26513408 i_blocks 53255 00000020:00200000:0.0:1789760772.455673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.455684:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.455720:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26513408, count: 4096 00000080:00200000:0.0:1789760772.455727:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26513408, bytes: 4096 00000080:00200000:0.0:1789760772.455737:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.455746:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.455749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26513408, 26517504) 0 0 00000080:00200000:0.0:1789760772.455753:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.455768:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6473, 6473] 00020000:00200000:0.0:1789760772.455776:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26513408 chunk: [26513408, 26517504) 26517504, 4096 00020000:00200000:0.0:1789760772.455781:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.455800:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6473, 6473] 00000008:00010000:0.0:1789760772.455829:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.455835:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.455847:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26513408, 26517504) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.455860:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26513408=0x1949000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.455915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.455918:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26517504 00000020:00010000:0.0:1789760772.455926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.455930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.455933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.455937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.455940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.455947:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.455955:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.455972:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760772.456077:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 26513408<>26517504 00000100:00100000:1.0:1789760772.456346:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813120:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760772.498378:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973792 gen 3 00000008:00010000:1.0:1789760772.498469:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760772.498478:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.498488:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.498511:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760772.498541:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813120:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.498782:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6473, 6473] 00000080:00200000:0.0:1789760772.498801:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.498807:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.498838:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26517504 i_blocks 53255 00000020:00200000:0.0:1789760772.498842:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.498853:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.498892:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26517504, count: 4096 00000080:00200000:0.0:1789760772.498900:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26517504, bytes: 4096 00000080:00200000:0.0:1789760772.498910:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.498919:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.498922:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26517504, 26521600) 0 0 00000080:00200000:0.0:1789760772.498927:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.498945:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6474, 6474] 00020000:00200000:0.0:1789760772.498955:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26517504 chunk: [26517504, 26521600) 26521600, 4096 00020000:00200000:0.0:1789760772.498960:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.498983:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6474, 6474] 00000008:00010000:0.0:1789760772.499004:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.499011:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.499025:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26517504, 26521600) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.499041:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26517504=0x194a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.499112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.499116:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26521600 00000020:00010000:0.0:1789760772.499125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.499129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.499132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.499136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.499139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.499147:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.499385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.499439:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760772.499553:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26517504<>26521600 00000100:00100000:1.0:1789760772.499674:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760772.499695:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813248:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760772.529792:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760772.529808:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.529820:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.529842:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760772.529873:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813248:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.529964:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6474, 6474] 00000080:00200000:0.0:1789760772.530077:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.530084:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.530108:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26521600 i_blocks 53255 00000020:00200000:0.0:1789760772.530112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.530123:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.530154:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26521600, count: 4096 00000080:00200000:0.0:1789760772.530162:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26521600, bytes: 4096 00000080:00200000:0.0:1789760772.530172:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.530181:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.530184:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26521600, 26525696) 0 0 00000080:00200000:0.0:1789760772.530189:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.530205:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6475, 6475] 00020000:00200000:0.0:1789760772.530213:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26521600 chunk: [26521600, 26525696) 26525696, 4096 00020000:00200000:0.0:1789760772.530218:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.530237:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6475, 6475] 00000008:00010000:0.0:1789760772.530254:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.530260:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.530272:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26521600, 26525696) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.530286:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26521600=0x194b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.530342:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.530345:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26525696 00000020:00010000:0.0:1789760772.530353:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.530356:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.530359:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.530445:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.530448:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.530457:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.530464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.530479:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760772.530588:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 26521600<>26525696 00000100:00100000:0.0:1789760772.530627:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813376:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.562050:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973794 gen 3 00000008:00010000:0.0:1789760772.562129:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760772.562138:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.562148:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.562445:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760772.562476:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813376:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760772.562587:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6475, 6475] 00000080:00200000:1.0:1789760772.562604:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.562610:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.562627:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26525696 i_blocks 53255 00000020:00200000:1.0:1789760772.562631:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.562642:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.562683:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26525696, count: 4096 00000080:00200000:1.0:1789760772.562691:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26525696, bytes: 4096 00000080:00200000:1.0:1789760772.562703:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.562996:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.562999:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26525696, 26529792) 0 0 00000080:00200000:1.0:1789760772.563005:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.563021:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6476, 6476] 00020000:00200000:1.0:1789760772.563030:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26525696 chunk: [26525696, 26529792) 26529792, 4096 00020000:00200000:1.0:1789760772.563035:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.563057:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6476, 6476] 00000008:00010000:1.0:1789760772.563076:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.563082:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760772.563094:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26525696, 26529792) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.563117:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26525696=0x194c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.563317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.563320:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26529792 00000020:00010000:1.0:1789760772.563330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.563334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.563338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.563341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.563345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.563356:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.563363:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.563381:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760772.563531:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 26525696<>26529792 00000100:00100000:0.0:1789760772.564093:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493813504:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.618778:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973795 gen 3 00000008:00010000:0.0:1789760772.618875:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760772.618884:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.618895:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.618918:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760772.618947:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493813504:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760772.619060:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6476, 6476] 00000080:00200000:1.0:1789760772.619078:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.619085:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.619114:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26529792 i_blocks 53255 00000020:00200000:1.0:1789760772.619118:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.619129:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.619285:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26529792, count: 4096 00000080:00200000:1.0:1789760772.619294:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26529792, bytes: 4096 00000080:00200000:1.0:1789760772.619304:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.619314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.619316:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26529792, 26533888) 0 0 00000080:00200000:1.0:1789760772.619321:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.619339:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6477, 6477] 00020000:00200000:1.0:1789760772.619348:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26529792 chunk: [26529792, 26533888) 26533888, 4096 00020000:00200000:1.0:1789760772.619352:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.619370:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6477, 6477] 00000008:00010000:1.0:1789760772.619388:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.619422:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760772.619435:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26529792, 26533888) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.619447:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26529792=0x194d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.619498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.619501:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26533888 00000020:00010000:1.0:1789760772.619508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.619511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.619514:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.619517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.619520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.619530:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.619536:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.619550:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760772.619786:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 26529792<>26533888 00000100:00100000:0.0:1789760772.619928:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813632:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.649657:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973796 gen 3 00000008:00010000:0.0:1789760772.649716:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760772.649724:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.649736:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.649807:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760772.649849:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493813632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760772.649963:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6477, 6477] 00000080:00200000:3.0:1789760772.649979:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.649985:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.650000:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26533888 i_blocks 53255 00000020:00200000:3.0:1789760772.650004:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.650014:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.650045:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26533888, count: 4096 00000080:00200000:3.0:1789760772.650051:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26533888, bytes: 4096 00000080:00200000:3.0:1789760772.650061:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.650147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.650213:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26533888, 26537984) 0 0 00000080:00200000:3.0:1789760772.650220:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.650237:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6478, 6478] 00020000:00200000:3.0:1789760772.650245:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26533888 chunk: [26533888, 26537984) 26537984, 4096 00020000:00200000:3.0:1789760772.650250:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.650269:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6478, 6478] 00000008:00010000:3.0:1789760772.650285:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.650291:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760772.650304:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26533888, 26537984) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.650319:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26533888=0x194e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.650376:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.650379:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26537984 00000020:00010000:3.0:1789760772.650388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.650391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.650473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.650478:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.650481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.650490:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.650498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.650515:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760772.650684:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 26533888<>26537984 00000100:00100000:3.0:1789760772.650729:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.650747:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493813760:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760772.654250:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493814016:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760772.654374:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1789760772.654381:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493813888:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760772.654437:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493814144:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760772.657859:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493814016:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760772.657900:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493813888:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760772.657913:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493814144:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760772.701529:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973797 gen 3 00000008:00010000:3.0:1789760772.701796:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760772.701806:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.701950:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.702049:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760772.702080:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493813760:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.702242:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6478, 6478] 00000080:00200000:2.0:1789760772.702259:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.702265:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.702290:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26537984 i_blocks 53255 00000020:00200000:2.0:1789760772.702295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.702305:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.702338:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26537984, count: 4096 00000080:00200000:2.0:1789760772.702345:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26537984, bytes: 4096 00000080:00200000:2.0:1789760772.702360:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.702369:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.702371:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26537984, 26542080) 0 0 00000080:00200000:2.0:1789760772.702375:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.702391:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6479, 6479] 00020000:00200000:2.0:1789760772.702434:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26537984 chunk: [26537984, 26542080) 26542080, 4096 00020000:00200000:2.0:1789760772.702439:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.702469:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6479, 6479] 00000008:00010000:2.0:1789760772.702487:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.702493:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.702505:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26537984, 26542080) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.702520:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26537984=0x194f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.702761:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.702765:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26542080 00000020:00010000:2.0:1789760772.702773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.702777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.702780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.702784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.702787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.702795:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.702801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.702946:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760772.703145:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26537984<>26542080 00000100:00100000:3.0:1789760772.703220:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493814272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760772.761793:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973798 gen 3 00000008:00010000:3.0:1789760772.761845:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760772.761855:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.761868:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.761892:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760772.761922:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493814272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.762173:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6479, 6479] 00000080:00200000:2.0:1789760772.762191:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.762197:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.762212:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26542080 i_blocks 53255 00000020:00200000:2.0:1789760772.762217:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.762227:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.762263:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26542080, count: 4096 00000080:00200000:2.0:1789760772.762271:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26542080, bytes: 4096 00000080:00200000:2.0:1789760772.762282:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.762292:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.762294:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26542080, 26546176) 0 0 00000080:00200000:2.0:1789760772.762299:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.762315:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6480, 6480] 00020000:00200000:2.0:1789760772.762324:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26542080 chunk: [26542080, 26546176) 26546176, 4096 00020000:00200000:2.0:1789760772.762329:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.762347:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6480, 6480] 00000008:00010000:2.0:1789760772.762364:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.762371:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.762382:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26542080, 26546176) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.762678:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26542080=0x1950000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.762744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.762747:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26546176 00000020:00010000:2.0:1789760772.762871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.762876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.762879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.762884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.762887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.762896:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.762903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.762920:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760772.763024:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26542080<>26546176 00000100:00100000:3.0:1789760772.763108:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814400:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760772.808493:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760772.808511:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.808525:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.808547:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760772.808579:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814400:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.808682:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6480, 6480] 00000080:00200000:2.0:1789760772.808697:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.808703:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.808717:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26546176 i_blocks 53255 00000020:00200000:2.0:1789760772.808720:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.808740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.808772:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26546176, count: 4096 00000080:00200000:2.0:1789760772.808779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26546176, bytes: 4096 00000080:00200000:2.0:1789760772.808789:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.808799:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.808801:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26546176, 26550272) 0 0 00000080:00200000:2.0:1789760772.808806:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.808822:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6481, 6481] 00020000:00200000:2.0:1789760772.808931:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26546176 chunk: [26546176, 26550272) 26550272, 4096 00020000:00200000:2.0:1789760772.808937:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.808960:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6481, 6481] 00000008:00010000:2.0:1789760772.808979:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.808985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.808996:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26546176, 26550272) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.809011:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26546176=0x1951000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.809072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.809075:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26550272 00000020:00010000:2.0:1789760772.809084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.809086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.809089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.809093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.809107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.809115:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.809123:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.809137:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760772.809239:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26546176<>26550272 00000100:00100000:3.0:1789760772.809303:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493814528:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760772.842804:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973800 gen 3 00000008:00010000:3.0:1789760772.844072:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760772.844081:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.844093:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.844307:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760772.844351:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493814528:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760772.845071:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6481, 6481] 00000080:00200000:0.0:1789760772.845090:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.845105:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.845122:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26550272 i_blocks 53255 00000020:00200000:0.0:1789760772.845127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.845139:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.845297:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26550272, count: 4096 00000080:00200000:0.0:1789760772.845306:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26550272, bytes: 4096 00000080:00200000:0.0:1789760772.845385:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.845466:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.845469:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26550272, 26554368) 0 0 00000080:00200000:0.0:1789760772.845474:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.845492:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6482, 6482] 00020000:00200000:0.0:1789760772.845500:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26550272 chunk: [26550272, 26554368) 26554368, 4096 00020000:00200000:0.0:1789760772.845504:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.845525:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6482, 6482] 00000008:00010000:0.0:1789760772.845543:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.845549:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760772.845561:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26550272, 26554368) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.845641:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26550272=0x1952000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.845707:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.845710:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26554368 00000020:00010000:0.0:1789760772.845719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.845722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.845725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.845729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.845732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.845740:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.845748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.845770:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760772.846147:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 26550272<>26554368 00000100:00100000:1.0:1789760772.846456:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493814656:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760772.880587:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973801 gen 3 00000008:00010000:1.0:1789760772.880645:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760772.880654:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.880666:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.880740:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760772.880777:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493814656:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760772.882808:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6482, 6482] 00000080:00200000:3.0:1789760772.882833:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.882840:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.882858:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26554368 i_blocks 53255 00000020:00200000:3.0:1789760772.882863:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.882879:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.882914:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26554368, count: 4096 00000080:00200000:3.0:1789760772.882923:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26554368, bytes: 4096 00000080:00200000:3.0:1789760772.882938:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.882948:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.882950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26554368, 26558464) 0 0 00000080:00200000:3.0:1789760772.882955:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.882974:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6483, 6483] 00020000:00200000:3.0:1789760772.882985:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26554368 chunk: [26554368, 26558464) 26558464, 4096 00020000:00200000:3.0:1789760772.882991:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.883347:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6483, 6483] 00000008:00010000:2.0:1789760772.883370:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.883376:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.883388:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26554368, 26558464) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.883420:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26554368=0x1953000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.883488:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.883491:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26558464 00000020:00010000:2.0:1789760772.883500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.883503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.883507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.883510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.883513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.883522:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.883529:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.883544:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760772.883659:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26554368<>26558464 00000100:00100000:3.0:1789760772.883779:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760772.935969:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973802 gen 3 00000008:00010000:3.0:1789760772.936022:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760772.936031:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.936042:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.936067:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760772.936109:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814784:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.936232:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6483, 6483] 00000080:00200000:2.0:1789760772.936249:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.936255:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.936270:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26558464 i_blocks 53255 00000020:00200000:2.0:1789760772.936274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.936283:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.936453:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26558464, count: 4096 00000080:00200000:2.0:1789760772.936462:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26558464, bytes: 4096 00000080:00200000:2.0:1789760772.936474:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.936484:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.936486:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26558464, 26562560) 0 0 00000080:00200000:2.0:1789760772.936491:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.936507:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6484, 6484] 00020000:00200000:2.0:1789760772.936516:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26558464 chunk: [26558464, 26562560) 26562560, 4096 00020000:00200000:2.0:1789760772.936520:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.936540:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6484, 6484] 00000008:00010000:2.0:1789760772.936557:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.936563:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.936586:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26558464, 26562560) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.936601:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26558464=0x1954000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.936654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.936657:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26562560 00000020:00010000:2.0:1789760772.936666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.936670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.936673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.936677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.936680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.936687:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.936694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.936709:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760772.936817:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26558464<>26562560 00000100:00100000:3.0:1789760772.938300:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.938336:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814912:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760772.976680:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973803 gen 3 00000008:00010000:3.0:1789760772.976733:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760772.976756:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.976768:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.976969:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6484, 6484] 00000008:00010000:3.0:1789760772.976974:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000080:00200000:2.0:1789760772.976987:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.976993:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760772.977004:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493814912:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760772.977008:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26562560 i_blocks 53255 00000020:00200000:2.0:1789760772.977011:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.977022:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.977068:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26562560, count: 4096 00000080:00200000:2.0:1789760772.977075:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26562560, bytes: 4096 00000080:00200000:2.0:1789760772.977084:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.977092:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.977104:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26562560, 26566656) 0 0 00000080:00200000:2.0:1789760772.977110:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.977126:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6485, 6485] 00020000:00200000:2.0:1789760772.977135:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26562560 chunk: [26562560, 26566656) 26566656, 4096 00020000:00200000:2.0:1789760772.977139:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.977159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6485, 6485] 00000008:00010000:2.0:1789760772.977177:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.977193:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760772.977206:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26562560, 26566656) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.977223:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26562560=0x1955000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.977279:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.977282:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26566656 00000020:00010000:2.0:1789760772.977291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.977295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.977298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.977301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.977304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.977311:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.977318:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.977332:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760772.977453:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26562560<>26566656 00000100:00100000:3.0:1789760772.977616:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493815040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.018847:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973804 gen 3 00000008:00010000:3.0:1789760773.018900:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760773.018908:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.018919:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.018991:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760773.019022:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493815040:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760773.019457:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6485, 6485] 00000080:00200000:2.0:1789760773.019472:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.019478:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.019494:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26566656 i_blocks 53255 00000020:00200000:2.0:1789760773.019498:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.019507:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.019536:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26566656, count: 4096 00000080:00200000:2.0:1789760773.019542:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26566656, bytes: 4096 00000080:00200000:2.0:1789760773.019553:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.019561:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.019563:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26566656, 26570752) 0 0 00000080:00200000:2.0:1789760773.019567:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.019792:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6486, 6486] 00020000:00200000:2.0:1789760773.019802:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26566656 chunk: [26566656, 26570752) 26570752, 4096 00020000:00200000:2.0:1789760773.019807:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.019826:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6486, 6486] 00000008:00010000:2.0:1789760773.019844:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.019851:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.019863:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26566656, 26570752) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.019879:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26566656=0x1956000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.019933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.019935:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26570752 00000020:00010000:2.0:1789760773.019945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.019948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.019951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.019954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.019957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.019964:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.019970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.019986:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760773.021447:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26566656<>26570752 00000100:00100000:3.0:1789760773.021526:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493815168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.080061:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973805 gen 3 00000008:00010000:3.0:1789760773.080122:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760773.080131:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.080143:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.080171:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760773.080202:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493815168:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.080318:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6486, 6486] 00000080:00200000:0.0:1789760773.080336:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.080342:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.080358:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26570752 i_blocks 53255 00000020:00200000:0.0:1789760773.080362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.080374:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.080448:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26570752, count: 4096 00000080:00200000:0.0:1789760773.080456:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26570752, bytes: 4096 00000080:00200000:0.0:1789760773.080569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.080641:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.080643:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26570752, 26574848) 0 0 00000080:00200000:0.0:1789760773.080649:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.080665:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6487, 6487] 00020000:00200000:0.0:1789760773.080672:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26570752 chunk: [26570752, 26574848) 26574848, 4096 00020000:00200000:0.0:1789760773.080677:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.080697:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6487, 6487] 00000008:00010000:0.0:1789760773.080716:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.080722:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.080736:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26570752, 26574848) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.080749:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26570752=0x1957000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.080809:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.080812:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26574848 00000020:00010000:0.0:1789760773.080892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.080896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.080899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.080902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.080905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.080913:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.080921:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.080938:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760773.081044:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 26570752<>26574848 00000100:00100000:1.0:1789760773.081140:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493815296:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.115112:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973806 gen 3 00000008:00010000:1.0:1789760773.115224:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760773.115233:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.115243:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.115326:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760773.115357:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493815296:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.115503:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6487, 6487] 00000080:00200000:0.0:1789760773.115520:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.115526:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.115559:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26574848 i_blocks 53255 00000020:00200000:0.0:1789760773.115563:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.115573:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.115605:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26574848, count: 4096 00000080:00200000:0.0:1789760773.115612:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26574848, bytes: 4096 00000080:00200000:0.0:1789760773.115622:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.115631:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.115634:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26574848, 26578944) 0 0 00000080:00200000:0.0:1789760773.115638:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.115653:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6488, 6488] 00020000:00200000:0.0:1789760773.115661:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26574848 chunk: [26574848, 26578944) 26578944, 4096 00020000:00200000:0.0:1789760773.115666:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.115685:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6488, 6488] 00000008:00010000:0.0:1789760773.115703:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.115709:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.115722:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26574848, 26578944) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.115809:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26574848=0x1958000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.115924:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.115927:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26578944 00000020:00010000:0.0:1789760773.115935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.115939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.115942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.115946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.115949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.115956:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.115962:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.115976:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760773.116075:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26574848<>26578944 00000100:00100000:1.0:1789760773.116144:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493815424:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.173194:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973807 gen 3 00000008:00010000:1.0:1789760773.173245:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760773.173253:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.173338:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.173361:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760773.173391:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493815424:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.173505:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6488, 6488] 00000080:00200000:0.0:1789760773.173523:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.173530:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.173546:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26578944 i_blocks 53255 00000020:00200000:0.0:1789760773.173550:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.173561:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.173606:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26578944, count: 4096 00000080:00200000:0.0:1789760773.173614:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26578944, bytes: 4096 00000080:00200000:0.0:1789760773.173624:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.173633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.173635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26578944, 26583040) 0 0 00000080:00200000:0.0:1789760773.173639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.173654:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6489, 6489] 00020000:00200000:0.0:1789760773.173663:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26578944 chunk: [26578944, 26583040) 26583040, 4096 00020000:00200000:0.0:1789760773.173668:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.173687:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6489, 6489] 00000008:00010000:0.0:1789760773.173704:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.173711:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.173723:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26578944, 26583040) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.173737:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26578944=0x1959000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.173858:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.173862:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26583040 00000020:00010000:0.0:1789760773.173871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.173875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.173878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.173881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.173884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.173891:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.173898:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760773.174107:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:1.0:1789760773.174212:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 26578944<>26583040 00000100:00100000:1.0:1789760773.174252:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760773.174718:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493815552:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.218652:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973808 gen 3 00000008:00010000:1.0:1789760773.218702:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760773.218711:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.218722:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.218746:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760773.218783:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493815552:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.219061:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6489, 6489] 00000080:00200000:0.0:1789760773.219079:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.219085:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.219109:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26583040 i_blocks 53255 00000020:00200000:0.0:1789760773.219115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.219126:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.219162:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26583040, count: 4096 00000080:00200000:0.0:1789760773.219170:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26583040, bytes: 4096 00000080:00200000:0.0:1789760773.219178:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.219187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.219189:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26583040, 26587136) 0 0 00000080:00200000:0.0:1789760773.219194:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.219209:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6490, 6490] 00020000:00200000:0.0:1789760773.219218:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26583040 chunk: [26583040, 26587136) 26587136, 4096 00020000:00200000:0.0:1789760773.219222:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.219243:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6490, 6490] 00000008:00010000:0.0:1789760773.219260:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.219267:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.219279:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26583040, 26587136) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.219295:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26583040=0x195a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.219354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.219357:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26587136 00000020:00010000:0.0:1789760773.219365:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.219368:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.219371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.219374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.219377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.219384:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.219391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.219444:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760773.219545:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 26583040<>26587136 00000100:00100000:0.0:1789760773.219631:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760773.219650:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493815680:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760773.263838:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973809 gen 3 00000008:00010000:0.0:1789760773.263887:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760773.263897:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760773.263907:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760773.263933:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760773.263964:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493815680:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760773.264087:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6490, 6490] 00000080:00200000:3.0:1789760773.264113:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.264121:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.264136:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26587136 i_blocks 53255 00000020:00200000:3.0:1789760773.264140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.264151:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.264250:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26587136, count: 4096 00000080:00200000:3.0:1789760773.264258:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26587136, bytes: 4096 00000080:00200000:3.0:1789760773.264270:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.264279:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.264281:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26587136, 26591232) 0 0 00000080:00200000:3.0:1789760773.264286:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.264302:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6491, 6491] 00020000:00200000:3.0:1789760773.264311:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26587136 chunk: [26587136, 26591232) 26591232, 4096 00020000:00200000:3.0:1789760773.264316:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.264335:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6491, 6491] 00000008:00010000:3.0:1789760773.264352:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.264359:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760773.264370:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26587136, 26591232) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.264383:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26587136=0x195b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.264501:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.264505:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26591232 00000020:00010000:3.0:1789760773.264513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.264517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.264520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.264524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.264527:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.264535:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.264542:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.264557:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760773.264809:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 26587136<>26591232 00000100:00100000:2.0:1789760773.264886:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493815808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.310152:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973810 gen 3 00000008:00010000:3.0:1789760773.310202:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760773.310212:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.310224:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.310282:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760773.310311:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493815808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760773.310495:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6491, 6491] 00000080:00200000:2.0:1789760773.310512:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.310519:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.310535:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26591232 i_blocks 53255 00000020:00200000:2.0:1789760773.310539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.310548:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.310638:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26591232, count: 4096 00000080:00200000:2.0:1789760773.310645:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26591232, bytes: 4096 00000080:00200000:2.0:1789760773.310656:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.310666:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.310668:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26591232, 26595328) 0 0 00000080:00200000:2.0:1789760773.310673:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.310691:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6492, 6492] 00020000:00200000:2.0:1789760773.310700:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26591232 chunk: [26591232, 26595328) 26595328, 4096 00020000:00200000:2.0:1789760773.310704:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.310724:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6492, 6492] 00000008:00010000:2.0:1789760773.310741:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.310747:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.310758:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26591232, 26595328) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.310826:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26591232=0x195c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.310953:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.310957:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26595328 00000020:00010000:2.0:1789760773.310966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.310969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.310971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.310975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.310978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.310986:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.310993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.311008:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760773.311177:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26591232<>26595328 00000100:00100000:3.0:1789760773.311242:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493815936:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760773.357206:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760773.369073:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973811 gen 3 00000008:00010000:3.0:1789760773.369136:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760773.369144:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.369156:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.369177:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760773.369205:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493815936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760773.369873:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6492, 6492] 00000080:00200000:2.0:1789760773.369891:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.369897:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.369914:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26595328 i_blocks 53255 00000020:00200000:2.0:1789760773.369918:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.369929:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.369965:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26595328, count: 4096 00000080:00200000:2.0:1789760773.369973:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26595328, bytes: 4096 00000080:00200000:2.0:1789760773.369984:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.369995:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.369997:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26595328, 26599424) 0 0 00000080:00200000:2.0:1789760773.370002:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.370018:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6493, 6493] 00020000:00200000:2.0:1789760773.370027:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26595328 chunk: [26595328, 26599424) 26599424, 4096 00020000:00200000:2.0:1789760773.370031:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.370051:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6493, 6493] 00000008:00010000:2.0:1789760773.370068:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.370074:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.370085:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26595328, 26599424) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.370107:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26595328=0x195d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.370164:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.370168:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26599424 00000020:00010000:2.0:1789760773.370176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.370180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.370183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.370186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.370190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.370198:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.370206:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.370222:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760773.370535:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26595328<>26599424 00000100:00100000:3.0:1789760773.370789:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493816064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.400065:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973812 gen 3 00000008:00010000:3.0:1789760773.400121:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760773.400129:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.400141:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.400252:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760773.400282:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493816064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760773.400361:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6493, 6493] 00000080:00200000:2.0:1789760773.400376:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.400382:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.400429:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26599424 i_blocks 53255 00000020:00200000:2.0:1789760773.400433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.400444:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.400479:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26599424, count: 4096 00000080:00200000:2.0:1789760773.400486:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26599424, bytes: 4096 00000080:00200000:2.0:1789760773.400495:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.400503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.400505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26599424, 26603520) 0 0 00000080:00200000:2.0:1789760773.400510:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.400525:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6494, 6494] 00020000:00200000:2.0:1789760773.400534:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26599424 chunk: [26599424, 26603520) 26603520, 4096 00020000:00200000:2.0:1789760773.400539:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.400558:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6494, 6494] 00000008:00010000:2.0:1789760773.400639:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.400645:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.400657:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26599424, 26603520) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.400672:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26599424=0x195e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.400726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.400729:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26603520 00000020:00010000:2.0:1789760773.400736:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.400739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.400743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.400746:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.400749:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.400757:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.400763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.400776:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760773.400930:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26599424<>26603520 00000100:00100000:3.0:1789760773.400994:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493816192:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.449359:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973813 gen 3 00000008:00010000:3.0:1789760773.449546:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760773.449555:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.449565:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.449642:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6494, 6494] 00000008:00010000:3.0:1789760773.449645:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000080:00200000:2.0:1789760773.449661:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.449667:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.449683:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26603520 i_blocks 53255 00000020:00200000:2.0:1789760773.449687:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.449699:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.449734:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26603520, count: 4096 00000080:00200000:2.0:1789760773.449742:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26603520, bytes: 4096 00000080:00200000:2.0:1789760773.449753:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.449762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.449765:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26603520, 26607616) 0 0 00000080:00200000:2.0:1789760773.449770:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.449786:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6495, 6495] 00020000:00200000:2.0:1789760773.449795:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26603520 chunk: [26603520, 26607616) 26607616, 4096 00020000:00200000:2.0:1789760773.449799:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.449902:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6495, 6495] 00000008:00010000:2.0:1789760773.449922:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.449927:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.449938:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26603520, 26607616) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.449951:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26603520=0x195f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.450015:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.450018:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26607616 00000020:00010000:2.0:1789760773.450026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.450029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.450032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.450036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.450039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.450045:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.450052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760773.450626:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493816192:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760773.456107:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760773.456537:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26603520<>26607616 00000100:00100000:3.0:1789760773.456610:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760773.456782:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493816320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.516735:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973814 gen 3 00000008:00010000:3.0:1789760773.516774:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760773.516780:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.516790:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.517500:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760773.517532:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493816320:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.517696:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6495, 6495] 00000080:00200000:0.0:1789760773.517709:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.517715:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.517727:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26607616 i_blocks 53255 00000020:00200000:0.0:1789760773.517730:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.517740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.517769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26607616, count: 4096 00000080:00200000:0.0:1789760773.517775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26607616, bytes: 4096 00000080:00200000:0.0:1789760773.517785:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.517792:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.517794:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26607616, 26611712) 0 0 00000080:00200000:0.0:1789760773.517797:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.517810:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6496, 6496] 00020000:00200000:0.0:1789760773.517817:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26607616 chunk: [26607616, 26611712) 26611712, 4096 00020000:00200000:0.0:1789760773.517821:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.517835:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6496, 6496] 00000008:00010000:0.0:1789760773.517850:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.517855:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.517864:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26607616, 26611712) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.517875:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26607616=0x1960000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.517923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.517926:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26611712 00000020:00010000:0.0:1789760773.517933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.517936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.517938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.517942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.517944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.517951:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.517956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.517973:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760773.518266:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 26607616<>26611712 00000100:00100000:1.0:1789760773.518358:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.548912:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973815 gen 3 00000008:00010000:1.0:1789760773.548949:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760773.548955:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.548963:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.548985:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760773.549011:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816448:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.549108:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6496, 6496] 00000080:00200000:0.0:1789760773.549121:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.549126:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.549138:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26611712 i_blocks 53255 00000020:00200000:0.0:1789760773.549141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.549150:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.549178:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26611712, count: 4096 00000080:00200000:0.0:1789760773.549184:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26611712, bytes: 4096 00000080:00200000:0.0:1789760773.549192:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.549200:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.549202:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26611712, 26615808) 0 0 00000080:00200000:0.0:1789760773.549205:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.549217:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6497, 6497] 00020000:00200000:0.0:1789760773.549223:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26611712 chunk: [26611712, 26615808) 26615808, 4096 00020000:00200000:0.0:1789760773.549226:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.549240:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6497, 6497] 00000008:00010000:0.0:1789760773.549253:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.549257:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.549266:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26611712, 26615808) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.549279:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26611712=0x1961000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.549327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.549330:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26615808 00000020:00010000:0.0:1789760773.549337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.549340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.549342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.549347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.549349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.549355:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.549362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.549377:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760773.549501:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 26611712<>26615808 00000100:00100000:1.0:1789760773.549567:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760773.549581:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816576:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760773.581147:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760773.581157:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.581165:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.581185:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760773.581210:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816576:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.581305:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6497, 6497] 00000080:00200000:0.0:1789760773.581318:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.581323:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.581335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26615808 i_blocks 53255 00000020:00200000:0.0:1789760773.581339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.581347:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.581373:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26615808, count: 4096 00000080:00200000:0.0:1789760773.581379:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26615808, bytes: 4096 00000080:00200000:0.0:1789760773.581387:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.581424:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.581426:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26615808, 26619904) 0 0 00000080:00200000:0.0:1789760773.581430:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.581443:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6498, 6498] 00020000:00200000:0.0:1789760773.581450:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26615808 chunk: [26615808, 26619904) 26619904, 4096 00020000:00200000:0.0:1789760773.581453:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.581469:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6498, 6498] 00000008:00010000:0.0:1789760773.581483:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.581487:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.581496:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26615808, 26619904) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.581506:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26615808=0x1962000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.581551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.581554:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26619904 00000020:00010000:0.0:1789760773.581560:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.581563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.581566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.581569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.581572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.581578:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.581583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.581598:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760773.581673:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 26615808<>26619904 00000100:00100000:1.0:1789760773.581767:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816704:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.619729:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973817 gen 3 00000008:00010000:1.0:1789760773.619775:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760773.619782:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.619790:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.619811:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760773.619836:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816704:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.619938:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6498, 6498] 00000080:00200000:0.0:1789760773.619950:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.619956:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.619967:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26619904 i_blocks 53255 00000020:00200000:0.0:1789760773.619970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.619979:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.620018:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26619904, count: 4096 00000080:00200000:0.0:1789760773.620025:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26619904, bytes: 4096 00000080:00200000:0.0:1789760773.620033:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.620040:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.620042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26619904, 26624000) 0 0 00000080:00200000:0.0:1789760773.620046:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.620059:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6499, 6499] 00020000:00200000:0.0:1789760773.620066:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26619904 chunk: [26619904, 26624000) 26624000, 4096 00020000:00200000:0.0:1789760773.620070:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.620083:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6499, 6499] 00000008:00010000:0.0:1789760773.620105:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.620110:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.620120:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26619904, 26624000) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.620131:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26619904=0x1963000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.620185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.620188:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26624000 00000020:00010000:0.0:1789760773.620194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.620197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.620200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.620204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.620207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.620213:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.620219:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.620234:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760773.620305:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 26619904<>26624000 00000100:00100000:1.0:1789760773.620513:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760773.620528:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816832:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.652848:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973818 gen 3 00000008:00010000:1.0:1789760773.652883:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760773.652889:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.652897:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.652916:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760773.652941:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816832:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.653240:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6499, 6499] 00000080:00200000:0.0:1789760773.653254:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.653259:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.653270:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26624000 i_blocks 53255 00000020:00200000:0.0:1789760773.653273:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.653282:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.653307:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26624000, count: 4096 00000080:00200000:0.0:1789760773.653313:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26624000, bytes: 4096 00000080:00200000:0.0:1789760773.653321:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.653328:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.653330:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26624000, 26628096) 0 0 00000080:00200000:0.0:1789760773.653334:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.653348:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6500, 6500] 00020000:00200000:0.0:1789760773.653354:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26624000 chunk: [26624000, 26628096) 26628096, 4096 00020000:00200000:0.0:1789760773.653358:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.653372:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6500, 6500] 00000008:00010000:0.0:1789760773.653385:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.653390:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.653427:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26624000, 26628096) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.653437:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26624000=0x1964000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.653484:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.653487:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26628096 00000020:00010000:0.0:1789760773.653493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.653496:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.653499:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.653503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.653505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.653511:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.653517:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.653531:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760773.653659:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 26624000<>26628096 00000100:00100000:1.0:1789760773.653717:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816960:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.686519:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973819 gen 3 00000008:00010000:1.0:1789760773.686555:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760773.686563:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.686570:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.686639:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760773.686664:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493816960:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.687185:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6500, 6500] 00000080:00200000:0.0:1789760773.687199:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.687204:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.687216:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26628096 i_blocks 53255 00000020:00200000:0.0:1789760773.687220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.687228:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.687256:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26628096, count: 4096 00000080:00200000:0.0:1789760773.687263:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26628096, bytes: 4096 00000080:00200000:0.0:1789760773.687271:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.687279:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.687280:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26628096, 26632192) 0 0 00000080:00200000:0.0:1789760773.687284:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.687297:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6501, 6501] 00020000:00200000:0.0:1789760773.687304:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26628096 chunk: [26628096, 26632192) 26632192, 4096 00020000:00200000:0.0:1789760773.687741:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.687761:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6501, 6501] 00000008:00010000:0.0:1789760773.687776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.687781:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.687789:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26628096, 26632192) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.687801:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26628096=0x1965000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.687852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.687855:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26632192 00000020:00010000:0.0:1789760773.687861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.687864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.687867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.687870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.687873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.687879:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.687885:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.687905:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760773.687985:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 26628096<>26632192 00000100:00100000:1.0:1789760773.688042:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760773.688059:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493817088:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.720857:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973820 gen 3 00000008:00010000:1.0:1789760773.720890:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760773.720896:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.720904:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.720923:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760773.720945:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493817088:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.721068:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6501, 6501] 00000080:00200000:0.0:1789760773.721080:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.721085:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.721106:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26632192 i_blocks 53255 00000020:00200000:0.0:1789760773.721109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.721119:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.721146:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26632192, count: 4096 00000080:00200000:0.0:1789760773.721152:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26632192, bytes: 4096 00000080:00200000:0.0:1789760773.721159:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.721166:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.721168:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26632192, 26636288) 0 0 00000080:00200000:0.0:1789760773.721172:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.721183:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6502, 6502] 00020000:00200000:0.0:1789760773.721190:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26632192 chunk: [26632192, 26636288) 26636288, 4096 00020000:00200000:0.0:1789760773.721193:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.721207:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6502, 6502] 00000008:00010000:0.0:1789760773.721220:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.721224:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.721232:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26632192, 26636288) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.721242:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26632192=0x1966000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.721288:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.721291:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26636288 00000020:00010000:0.0:1789760773.721298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.721301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.721303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.721307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.721309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.721316:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.721321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.721333:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760773.721445:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26632192<>26636288 00000100:00100000:0.0:1789760773.721471:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760773.721482:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817216:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760773.763843:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973821 gen 3 00000008:00010000:0.0:1789760773.763876:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760773.763883:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760773.763891:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760773.763911:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760773.763934:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817216:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760773.764145:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6502, 6502] 00000080:00200000:3.0:1789760773.764157:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.764162:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760773.764173:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26636288 i_blocks 53255 00000020:00200000:3.0:1789760773.764176:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760773.764185:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760773.764211:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26636288, count: 4096 00000080:00200000:3.0:1789760773.764218:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26636288, bytes: 4096 00000080:00200000:3.0:1789760773.764227:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760773.764234:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760773.764236:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26636288, 26640384) 0 0 00000080:00200000:3.0:1789760773.764240:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760773.764252:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6503, 6503] 00020000:00200000:3.0:1789760773.764259:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26636288 chunk: [26636288, 26640384) 26640384, 4096 00020000:00200000:3.0:1789760773.764262:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760773.764279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6503, 6503] 00000008:00010000:3.0:1789760773.764292:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760773.764296:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760773.764305:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26636288, 26640384) append 0 append lockpos 0 00000080:00200000:3.0:1789760773.764316:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26636288=0x1967000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760773.764358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760773.764360:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26640384 00000020:00010000:3.0:1789760773.764367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760773.764371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.764374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760773.764377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760773.764379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760773.764385:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760773.764391:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760773.765144:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760773.765233:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 26636288<>26640384 00000100:00100000:2.0:1789760773.765303:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493817344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760773.830438:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973822 gen 3 00000008:00010000:3.0:1789760773.830473:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760773.830480:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.830488:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.830499:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760773.830531:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493817344:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.831461:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6503, 6503] 00000080:00200000:0.0:1789760773.831477:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.831483:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.831497:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26640384 i_blocks 53255 00000020:00200000:0.0:1789760773.831501:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.831511:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.831551:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26640384, count: 4096 00000080:00200000:0.0:1789760773.831558:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26640384, bytes: 4096 00000080:00200000:0.0:1789760773.831569:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.831577:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.831579:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26640384, 26644480) 0 0 00000080:00200000:0.0:1789760773.831583:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.831595:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6504, 6504] 00020000:00200000:0.0:1789760773.831602:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26640384 chunk: [26640384, 26644480) 26644480, 4096 00020000:00200000:0.0:1789760773.831606:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.831623:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6504, 6504] 00000008:00010000:0.0:1789760773.831638:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.831644:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.831653:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26640384, 26644480) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.831664:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26640384=0x1968000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.831717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.831719:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26644480 00000020:00010000:0.0:1789760773.831726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.831730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.831732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.831735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.831738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.831744:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.831750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.831767:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760773.831848:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26640384<>26644480 00000100:00100000:1.0:1789760773.831924:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817472:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.899998:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973823 gen 3 00000008:00010000:1.0:1789760773.900035:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760773.900042:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.900052:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.900062:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760773.900098:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817472:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.900235:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6504, 6504] 00000080:00200000:0.0:1789760773.900249:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.900256:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.900277:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26644480 i_blocks 53255 00000020:00200000:0.0:1789760773.900281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.900290:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.900327:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26644480, count: 4096 00000080:00200000:0.0:1789760773.900333:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26644480, bytes: 4096 00000080:00200000:0.0:1789760773.900340:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.900347:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.900349:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26644480, 26648576) 0 0 00000080:00200000:0.0:1789760773.900354:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.900365:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6505, 6505] 00020000:00200000:0.0:1789760773.900381:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26644480 chunk: [26644480, 26648576) 26648576, 4096 00020000:00200000:0.0:1789760773.900385:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.900426:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6505, 6505] 00000008:00010000:0.0:1789760773.900441:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.900445:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.900455:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26644480, 26648576) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.900467:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26644480=0x1969000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.900513:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.900516:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26648576 00000020:00010000:0.0:1789760773.901047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.901051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.901054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.901057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.901060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.901067:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.901073:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.901099:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760773.901176:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 26644480<>26648576 00000100:00100000:1.0:1789760773.902844:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493817600:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760773.944023:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760773.944034:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.944043:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.944135:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760773.944169:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493817600:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760773.944288:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6505, 6505] 00000080:00200000:0.0:1789760773.944302:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760773.944306:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760773.944317:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26648576 i_blocks 53255 00000020:00200000:0.0:1789760773.944321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760773.944330:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760773.944359:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26648576, count: 4096 00000080:00200000:0.0:1789760773.944365:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26648576, bytes: 4096 00000080:00200000:0.0:1789760773.944372:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760773.944380:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760773.944382:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26648576, 26652672) 0 0 00000080:00200000:0.0:1789760773.944386:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760773.944427:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6506, 6506] 00020000:00200000:0.0:1789760773.944435:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26648576 chunk: [26648576, 26652672) 26652672, 4096 00020000:00200000:0.0:1789760773.944438:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760773.944454:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6506, 6506] 00000008:00010000:0.0:1789760773.944468:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760773.944472:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760773.944482:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26648576, 26652672) append 0 append lockpos 0 00000080:00200000:0.0:1789760773.944493:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26648576=0x196a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760773.944541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760773.944543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26652672 00000020:00010000:0.0:1789760773.944550:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760773.944553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.944556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760773.944559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760773.944562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760773.944568:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760773.944573:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760773.944587:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760773.944669:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 26648576<>26652672 00000100:00100000:1.0:1789760773.944738:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817728:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760773.982038:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973824 gen 3 00000100:00100000:1.0:1789760773.982083:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304f480 x1876699493817728/t4294973825(4294973825) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760789 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760773.982138:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760773.982143:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760773.982151:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760773.982182:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760773.982205:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493817728:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760773.982272:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6506, 6506] 00000080:00200000:2.0:1789760773.982286:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.982291:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.982304:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26652672 i_blocks 53255 00000020:00200000:2.0:1789760773.982307:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.982317:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.982344:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26652672, count: 4096 00000080:00200000:2.0:1789760773.982349:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26652672, bytes: 4096 00000080:00200000:2.0:1789760773.982360:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.982367:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.982369:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26652672, 26656768) 0 0 00000080:00200000:2.0:1789760773.982373:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.982386:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6507, 6507] 00020000:00200000:2.0:1789760773.982429:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26652672 chunk: [26652672, 26656768) 26656768, 4096 00020000:00200000:2.0:1789760773.982433:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.982451:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6507, 6507] 00000008:00010000:2.0:1789760773.982465:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.982470:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760773.982479:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26652672, 26656768) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.982491:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26652672=0x196b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.982534:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.982537:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26656768 00000020:00010000:2.0:1789760773.982543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.982547:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.982549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.982553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.982556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.982562:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.982578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.982593:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760773.982667:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26652672<>26656768 00000100:00100000:3.0:1789760773.982771:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493817856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760774.025606:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973826 gen 3 00000008:00010000:3.0:1789760774.025653:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760774.025659:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.025667:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.025687:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760774.025712:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493817856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760774.025804:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6507, 6507] 00000080:00200000:2.0:1789760774.025817:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.025822:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.025833:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26656768 i_blocks 53255 00000020:00200000:2.0:1789760774.025836:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.025845:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.025871:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26656768, count: 4096 00000080:00200000:2.0:1789760774.025877:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26656768, bytes: 4096 00000080:00200000:2.0:1789760774.025885:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.025892:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.025894:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26656768, 26660864) 0 0 00000080:00200000:2.0:1789760774.025898:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.025911:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6508, 6508] 00020000:00200000:2.0:1789760774.025917:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26656768 chunk: [26656768, 26660864) 26660864, 4096 00020000:00200000:2.0:1789760774.025921:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.025937:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6508, 6508] 00000008:00010000:2.0:1789760774.025950:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.025955:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760774.025965:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26656768, 26660864) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.025976:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26656768=0x196c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.026020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.026023:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26660864 00000020:00010000:2.0:1789760774.026030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.026033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.026035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.026039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.026041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.026047:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.026053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.026067:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760774.026160:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26656768<>26660864 00000100:00100000:3.0:1789760774.026286:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.026299:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493817984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760774.066414:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973827 gen 3 00000008:00010000:3.0:1789760774.066451:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760774.066458:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.066467:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.066485:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760774.066511:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493817984:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760774.066907:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6508, 6508] 00000080:00200000:2.0:1789760774.066921:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.066927:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.066940:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26660864 i_blocks 53255 00000020:00200000:2.0:1789760774.066944:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.066953:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.066982:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26660864, count: 4096 00000080:00200000:2.0:1789760774.066987:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26660864, bytes: 4096 00000080:00200000:2.0:1789760774.066996:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.067003:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.067005:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26660864, 26664960) 0 0 00000080:00200000:2.0:1789760774.067008:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.067020:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6509, 6509] 00020000:00200000:2.0:1789760774.067027:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26660864 chunk: [26660864, 26664960) 26664960, 4096 00020000:00200000:2.0:1789760774.067031:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.067046:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6509, 6509] 00000008:00010000:2.0:1789760774.067059:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.067064:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760774.067074:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26660864, 26664960) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.067084:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26660864=0x196d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.067147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.067150:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26664960 00000020:00010000:2.0:1789760774.067157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.067161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.067163:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.067166:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.067169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.067176:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.067181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.067196:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760774.067273:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26660864<>26664960 00000100:00100000:3.0:1789760774.067374:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493818112:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760774.103085:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760774.103104:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.103113:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.103129:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760774.103219:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493818112:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760774.103307:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6509, 6509] 00000080:00200000:2.0:1789760774.103320:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.103325:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.103338:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26664960 i_blocks 53255 00000020:00200000:2.0:1789760774.103342:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.103351:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.103380:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26664960, count: 4096 00000080:00200000:2.0:1789760774.103385:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26664960, bytes: 4096 00000080:00200000:2.0:1789760774.103422:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.103430:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.103432:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26664960, 26669056) 0 0 00000080:00200000:2.0:1789760774.103436:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.103493:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6510, 6510] 00020000:00200000:2.0:1789760774.103500:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26664960 chunk: [26664960, 26669056) 26669056, 4096 00020000:00200000:2.0:1789760774.103504:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.103517:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6510, 6510] 00000008:00010000:2.0:1789760774.103531:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.103535:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760774.103545:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26664960, 26669056) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.103556:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26664960=0x196e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.103643:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.103645:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26669056 00000020:00010000:2.0:1789760774.103651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.103653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.103655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.103657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.103659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.103664:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.103667:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.103677:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760774.103739:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26664960<>26669056 00000100:00100000:3.0:1789760774.103783:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493818240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760774.149354:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973829 gen 3 00000008:00010000:3.0:1789760774.149430:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760774.149437:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.149446:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.149472:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760774.149496:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493818240:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.149691:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6510, 6510] 00000080:00200000:0.0:1789760774.149714:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.149719:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.149730:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26669056 i_blocks 53255 00000020:00200000:0.0:1789760774.149733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.149742:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.149769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26669056, count: 4096 00000080:00200000:0.0:1789760774.149775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26669056, bytes: 4096 00000080:00200000:0.0:1789760774.149783:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.149790:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.149792:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26669056, 26673152) 0 0 00000080:00200000:0.0:1789760774.149795:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.149807:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6511, 6511] 00020000:00200000:0.0:1789760774.149814:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26669056 chunk: [26669056, 26673152) 26673152, 4096 00020000:00200000:0.0:1789760774.149817:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.149830:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6511, 6511] 00000008:00010000:0.0:1789760774.149842:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.149846:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.149855:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26669056, 26673152) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.149865:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26669056=0x196f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.149909:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.149912:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26673152 00000020:00010000:0.0:1789760774.149918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.149921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.149924:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.149927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.149930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.149936:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.149942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.150246:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760774.150326:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 26669056<>26673152 00000100:00100000:1.0:1789760774.150459:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.198169:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973830 gen 3 00000008:00010000:1.0:1789760774.198203:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760774.198209:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.198217:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.198237:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760774.198264:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818368:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.198370:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6511, 6511] 00000080:00200000:0.0:1789760774.198383:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.198388:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.198435:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26673152 i_blocks 53255 00000020:00200000:0.0:1789760774.198439:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.198448:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.198474:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26673152, count: 4096 00000080:00200000:0.0:1789760774.198480:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26673152, bytes: 4096 00000080:00200000:0.0:1789760774.198488:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.198495:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.198497:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26673152, 26677248) 0 0 00000080:00200000:0.0:1789760774.198501:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.198514:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6512, 6512] 00020000:00200000:0.0:1789760774.198520:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26673152 chunk: [26673152, 26677248) 26677248, 4096 00020000:00200000:0.0:1789760774.198524:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.198538:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6512, 6512] 00000008:00010000:0.0:1789760774.198552:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.198557:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.198567:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26673152, 26677248) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.198578:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26673152=0x1970000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.198620:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.198622:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26677248 00000020:00010000:0.0:1789760774.198629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.198632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.198635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.198638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.198641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.198646:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.198651:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.198666:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760774.198754:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26673152<>26677248 00000100:00100000:1.0:1789760774.198861:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760774.198874:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818496:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.248183:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973831 gen 3 00000008:00010000:1.0:1789760774.248223:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760774.248229:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.248238:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.248267:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760774.248292:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818496:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.248533:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6512, 6512] 00000080:00200000:0.0:1789760774.248546:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.248550:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.248560:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26677248 i_blocks 53255 00000020:00200000:0.0:1789760774.248564:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.248572:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.248598:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26677248, count: 4096 00000080:00200000:0.0:1789760774.248603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26677248, bytes: 4096 00000080:00200000:0.0:1789760774.248611:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.248619:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.248621:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26677248, 26681344) 0 0 00000080:00200000:0.0:1789760774.248625:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.248637:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6513, 6513] 00020000:00200000:0.0:1789760774.248644:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26677248 chunk: [26677248, 26681344) 26681344, 4096 00020000:00200000:0.0:1789760774.248647:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.248662:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6513, 6513] 00000008:00010000:0.0:1789760774.248676:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.248681:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.248690:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26677248, 26681344) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.248700:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26677248=0x1971000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.248744:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.248747:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26681344 00000020:00010000:0.0:1789760774.248753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.248757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.248760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.248763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.248765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.248772:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.248778:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.248793:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760774.248873:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 26677248<>26681344 00000100:00100000:1.0:1789760774.248949:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818624:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.294064:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973832 gen 3 00000008:00010000:1.0:1789760774.294109:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760774.294116:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.294124:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.294267:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760774.294293:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493818624:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.298611:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6513, 6513] 00000080:00200000:0.0:1789760774.298631:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.298637:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.298676:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26681344 i_blocks 53255 00000020:00200000:0.0:1789760774.298681:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.298695:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.298722:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26681344, count: 4096 00000080:00200000:0.0:1789760774.298728:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26681344, bytes: 4096 00000080:00200000:0.0:1789760774.298740:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.298747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.298749:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26681344, 26685440) 0 0 00000080:00200000:0.0:1789760774.298753:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.298765:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6514, 6514] 00020000:00200000:0.0:1789760774.298772:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26681344 chunk: [26681344, 26685440) 26685440, 4096 00020000:00200000:0.0:1789760774.298776:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.298792:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6514, 6514] 00000008:00010000:0.0:1789760774.298806:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.298974:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.298991:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26681344, 26685440) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.299003:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26681344=0x1972000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.299054:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.299057:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26685440 00000020:00010000:0.0:1789760774.299064:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.299068:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.299071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f973f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.299074:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.299076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.299083:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.299098:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.299115:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760774.299195:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26681344<>26685440 00000100:00100000:1.0:1789760774.299280:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493818752:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.350706:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973833 gen 3 00000008:00010000:1.0:1789760774.350741:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760774.350747:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.350756:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.350768:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760774.350796:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493818752:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.350828:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6514, 6514] 00000080:00200000:0.0:1789760774.350850:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.350857:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.350873:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26685440 i_blocks 53255 00000020:00200000:0.0:1789760774.350881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.351002:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.351043:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26685440, count: 4096 00000080:00200000:1.0:1789760774.351050:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26685440, bytes: 4096 00000080:00200000:1.0:1789760774.351062:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.351070:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.351072:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26685440, 26689536) 0 0 00000080:00200000:1.0:1789760774.351077:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.351098:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6515, 6515] 00020000:00200000:1.0:1789760774.351105:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26685440 chunk: [26685440, 26689536) 26689536, 4096 00020000:00200000:1.0:1789760774.351109:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.351125:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6515, 6515] 00000008:00010000:1.0:1789760774.351139:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.351143:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760774.351152:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26685440, 26689536) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.351163:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26685440=0x1973000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.351207:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.351209:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26689536 00000020:00010000:1.0:1789760774.351216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.351219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.351222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f975a8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.351225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.351227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.351235:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.351241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.351277:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760774.351354:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 26685440<>26689536 00000100:00100000:1.0:1789760774.351446:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.351458:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493818880:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760774.380989:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760774.389968:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760774.389979:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.389987:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.390160:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760774.390187:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493818880:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.390467:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6515, 6515] 00000080:00200000:0.0:1789760774.390480:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.390485:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.390497:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26689536 i_blocks 53255 00000020:00200000:0.0:1789760774.390500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.390509:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.390596:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26689536, count: 4096 00000080:00200000:0.0:1789760774.390603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26689536, bytes: 4096 00000080:00200000:0.0:1789760774.390612:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.390620:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.390622:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26689536, 26693632) 0 0 00000080:00200000:0.0:1789760774.390626:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.390641:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6516, 6516] 00020000:00200000:0.0:1789760774.390648:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26689536 chunk: [26689536, 26693632) 26693632, 4096 00020000:00200000:0.0:1789760774.390653:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.390668:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6516, 6516] 00000008:00010000:0.0:1789760774.390684:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.390689:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.390698:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26689536, 26693632) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.390707:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26689536=0x1974000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.391218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.391222:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26693632 00000020:00010000:0.0:1789760774.391230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.391233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.391236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f975a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.391240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.391242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.391253:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.391259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.391280:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760774.391370:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 26689536<>26693632 00000100:00100000:0.0:1789760774.391453:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.391469:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819008:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760774.422219:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973835 gen 3 00000008:00010000:0.0:1789760774.422261:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760774.422266:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.422272:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.422288:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760774.422312:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819008:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760774.422321:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6516, 6516] 00000080:00200000:3.0:1789760774.422334:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760774.422339:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760774.422350:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26693632 i_blocks 53255 00000020:00200000:3.0:1789760774.422353:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760774.422362:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760774.422390:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26693632, count: 4096 00000080:00200000:3.0:1789760774.422424:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26693632, bytes: 4096 00000080:00200000:3.0:1789760774.423095:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760774.423105:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760774.423107:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26693632, 26697728) 0 0 00000080:00200000:3.0:1789760774.423112:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760774.423127:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6517, 6517] 00020000:00200000:3.0:1789760774.423133:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26693632 chunk: [26693632, 26697728) 26697728, 4096 00020000:00200000:3.0:1789760774.423137:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760774.423153:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6517, 6517] 00000008:00010000:3.0:1789760774.423169:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760774.423174:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760774.423184:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26693632, 26697728) append 0 append lockpos 0 00000080:00200000:3.0:1789760774.423196:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26693632=0x1975000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760774.423243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760774.423245:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26697728 00000020:00010000:3.0:1789760774.423252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760774.423255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.423257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f975a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760774.423260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760774.423263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760774.423268:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760774.423274:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760774.423294:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760774.423368:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 26693632<>26697728 00000100:00100000:2.0:1789760774.423519:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493819136:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760774.463361:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760774.463374:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.463382:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.463476:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760774.463500:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493819136:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.463595:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6517, 6517] 00000080:00200000:0.0:1789760774.463612:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.463617:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.463632:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26697728 i_blocks 53255 00000020:00200000:0.0:1789760774.463635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.463643:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.463680:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26697728, count: 4096 00000080:00200000:0.0:1789760774.463687:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26697728, bytes: 4096 00000080:00200000:0.0:1789760774.463694:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.463701:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.463703:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26697728, 26701824) 0 0 00000080:00200000:0.0:1789760774.463707:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.463720:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6518, 6518] 00020000:00200000:0.0:1789760774.463727:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26697728 chunk: [26697728, 26701824) 26701824, 4096 00020000:00200000:0.0:1789760774.463881:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.463902:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6518, 6518] 00000008:00010000:0.0:1789760774.463919:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.463925:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.463937:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26697728, 26701824) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.463950:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26697728=0x1976000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.464007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.464010:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26701824 00000020:00010000:0.0:1789760774.464019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.464022:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.464025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f975a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.464029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.464031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.464039:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.464046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.464063:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760774.464240:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 26697728<>26701824 00000100:00100000:1.0:1789760774.464325:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819264:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.498025:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973837 gen 3 00000008:00010000:1.0:1789760774.498103:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760774.498111:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.498122:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.498197:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760774.498226:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819264:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.501003:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6518, 6518] 00000080:00200000:0.0:1789760774.501024:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.501030:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.501168:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26701824 i_blocks 53255 00000020:00200000:1.0:1789760774.501175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.501200:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.501233:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26701824, count: 4096 00000080:00200000:1.0:1789760774.501240:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26701824, bytes: 4096 00000080:00200000:1.0:1789760774.501254:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.501263:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.501265:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26701824, 26705920) 0 0 00000080:00200000:1.0:1789760774.501269:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.501285:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6519, 6519] 00020000:00200000:1.0:1789760774.501294:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26701824 chunk: [26701824, 26705920) 26705920, 4096 00020000:00200000:1.0:1789760774.501299:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.501320:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6519, 6519] 00000008:00010000:1.0:1789760774.501341:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.501347:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760774.501358:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26701824, 26705920) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.501373:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26701824=0x1977000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.501461:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.501464:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26705920 00000020:00010000:1.0:1789760774.501473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.501477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.501480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.501484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.501487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.501499:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.501506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.501597:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760774.501702:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 26701824<>26705920 00000100:00100000:1.0:1789760774.501746:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760774.501770:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819392:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.535560:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973838 gen 3 00000008:00010000:1.0:1789760774.535612:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760774.535621:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.535632:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.535690:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760774.535720:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819392:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.535794:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6519, 6519] 00000080:00200000:0.0:1789760774.535812:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.535827:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.535843:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26705920 i_blocks 53255 00000020:00200000:0.0:1789760774.535848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.535859:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.535893:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26705920, count: 4096 00000080:00200000:0.0:1789760774.535900:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26705920, bytes: 4096 00000080:00200000:0.0:1789760774.535911:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.535920:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.535922:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26705920, 26710016) 0 0 00000080:00200000:0.0:1789760774.535927:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.535944:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6520, 6520] 00020000:00200000:0.0:1789760774.535953:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26705920 chunk: [26705920, 26710016) 26710016, 4096 00020000:00200000:0.0:1789760774.535958:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.535979:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6520, 6520] 00000008:00010000:0.0:1789760774.535998:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.536004:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.536017:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26705920, 26710016) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.536031:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26705920=0x1978000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.536103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.536106:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26710016 00000020:00010000:0.0:1789760774.536115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.536119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.536122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.536126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.536129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.536193:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.536201:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.536219:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760774.536328:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 26705920<>26710016 00000100:00100000:1.0:1789760774.536429:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819520:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.575390:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973839 gen 3 00000008:00010000:1.0:1789760774.575468:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760774.575477:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.575488:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.575553:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760774.575585:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493819520:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760774.576203:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6520, 6520] 00000080:00200000:2.0:1789760774.576222:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.576229:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.576247:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26710016 i_blocks 53255 00000020:00200000:2.0:1789760774.576252:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.576264:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.576299:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26710016, count: 4096 00000080:00200000:2.0:1789760774.576306:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26710016, bytes: 4096 00000080:00200000:2.0:1789760774.576318:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.576328:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.576330:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26710016, 26714112) 0 0 00000080:00200000:2.0:1789760774.576336:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.576352:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6521, 6521] 00020000:00200000:2.0:1789760774.576362:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26710016 chunk: [26710016, 26714112) 26714112, 4096 00020000:00200000:2.0:1789760774.576367:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.576389:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6521, 6521] 00000008:00010000:2.0:1789760774.576435:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.576441:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760774.576452:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26710016, 26714112) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.576464:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26710016=0x1979000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.576509:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.576511:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26714112 00000020:00010000:2.0:1789760774.576517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.576520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.576522:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.576524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.576526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.576532:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.576537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.576552:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760774.576664:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26710016<>26714112 00000100:00100000:3.0:1789760774.576803:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493819648:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760774.607669:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973840 gen 3 00000008:00010000:3.0:1789760774.607719:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760774.607726:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.607737:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.607753:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000080:00200000:2.0:1789760774.607777:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6521, 6521] 00000100:00100000:3.0:1789760774.607781:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493819648:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760774.607795:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.607800:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.607897:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26714112 i_blocks 53255 00000020:00200000:2.0:1789760774.607902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.607914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.607950:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26714112, count: 4096 00000080:00200000:2.0:1789760774.607957:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26714112, bytes: 4096 00000080:00200000:2.0:1789760774.607965:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.607981:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.607982:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26714112, 26718208) 0 0 00000080:00200000:2.0:1789760774.607987:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.608001:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6522, 6522] 00020000:00200000:2.0:1789760774.608008:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26714112 chunk: [26714112, 26718208) 26718208, 4096 00020000:00200000:2.0:1789760774.608011:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.608027:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6522, 6522] 00000008:00010000:2.0:1789760774.608042:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.608046:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760774.608057:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26714112, 26718208) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.608075:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26714112=0x197a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.608333:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.608336:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26718208 00000020:00010000:2.0:1789760774.608343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.608346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.608348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.608350:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.608352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.608359:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.608364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.608376:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760774.608502:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26714112<>26718208 00000100:00100000:2.0:1789760774.608543:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493819776:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760774.643625:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973841 gen 3 00000008:00010000:3.0:1789760774.643675:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760774.643684:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.643696:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.643708:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760774.643745:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493819776:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.643953:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6522, 6522] 00000080:00200000:0.0:1789760774.643970:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.643976:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.643991:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26718208 i_blocks 53255 00000020:00200000:0.0:1789760774.643994:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.644006:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.644037:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26718208, count: 4096 00000080:00200000:0.0:1789760774.644044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26718208, bytes: 4096 00000080:00200000:0.0:1789760774.644054:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.644062:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.644065:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26718208, 26722304) 0 0 00000080:00200000:0.0:1789760774.644069:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.644097:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6523, 6523] 00020000:00200000:0.0:1789760774.644106:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26718208 chunk: [26718208, 26722304) 26722304, 4096 00020000:00200000:0.0:1789760774.644111:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.644131:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6523, 6523] 00000008:00010000:0.0:1789760774.644157:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.644163:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.644176:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26718208, 26722304) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.644189:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26718208=0x197b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.644255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.644258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26722304 00000020:00010000:0.0:1789760774.644266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.644269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.644273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.644276:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.644279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.644288:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.644295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.644324:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760774.644471:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 26718208<>26722304 00000100:00100000:1.0:1789760774.644726:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493819904:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760774.680651:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760774.680667:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.680679:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.680704:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760774.680735:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493819904:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.681922:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6523, 6523] 00000080:00200000:0.0:1789760774.682646:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.682655:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.682673:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26722304 i_blocks 53255 00000020:00200000:0.0:1789760774.682678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.682692:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.683188:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26722304, count: 4096 00000080:00200000:0.0:1789760774.683242:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26722304, bytes: 4096 00000080:00200000:0.0:1789760774.683257:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.683265:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.683268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26722304, 26726400) 0 0 00000080:00200000:0.0:1789760774.683273:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.683288:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6524, 6524] 00020000:00200000:0.0:1789760774.683297:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26722304 chunk: [26722304, 26726400) 26726400, 4096 00020000:00200000:0.0:1789760774.683301:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.683320:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6524, 6524] 00000008:00010000:0.0:1789760774.683339:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.683344:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.683355:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26722304, 26726400) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.683369:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26722304=0x197c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.683456:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.683468:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26726400 00000020:00010000:0.0:1789760774.684571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.684581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.684585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.684589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.684593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.684606:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.684617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.684645:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760774.684762:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 26722304<>26726400 00000100:00100000:1.0:1789760774.684885:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493820032:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.729132:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973843 gen 3 00000008:00010000:1.0:1789760774.729200:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760774.729209:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.729218:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.729243:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760774.729272:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493820032:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.729381:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6524, 6524] 00000080:00200000:0.0:1789760774.729428:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.729435:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.729450:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26726400 i_blocks 53255 00000020:00200000:0.0:1789760774.729454:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.729464:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.729499:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26726400, count: 4096 00000080:00200000:0.0:1789760774.729506:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26726400, bytes: 4096 00000080:00200000:0.0:1789760774.729516:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.729524:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.729526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26726400, 26730496) 0 0 00000080:00200000:0.0:1789760774.729530:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.729546:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6525, 6525] 00020000:00200000:0.0:1789760774.729555:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26726400 chunk: [26726400, 26730496) 26730496, 4096 00020000:00200000:0.0:1789760774.729560:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.729580:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6525, 6525] 00000008:00010000:0.0:1789760774.729597:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.729603:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.729614:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26726400, 26730496) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.729627:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26726400=0x197d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.729682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.729685:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26730496 00000020:00010000:0.0:1789760774.729693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.729697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.729700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.729704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.729707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.729716:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.729723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.729739:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760774.729834:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 26726400<>26730496 00000100:00100000:1.0:1789760774.729936:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820160:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.763064:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973844 gen 3 00000008:00010000:1.0:1789760774.763133:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760774.763142:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.763153:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.763230:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760774.763262:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820160:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.765811:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6525, 6525] 00000080:00200000:0.0:1789760774.765847:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.765854:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.765873:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26730496 i_blocks 53255 00000020:00200000:0.0:1789760774.765879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.765895:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.765931:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26730496, count: 4096 00000080:00200000:0.0:1789760774.765939:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26730496, bytes: 4096 00000080:00200000:0.0:1789760774.765955:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.765966:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.765968:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26730496, 26734592) 0 0 00000080:00200000:0.0:1789760774.765977:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.765994:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6526, 6526] 00020000:00200000:0.0:1789760774.766003:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26730496 chunk: [26730496, 26734592) 26734592, 4096 00020000:00200000:0.0:1789760774.766008:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.766031:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6526, 6526] 00000008:00010000:0.0:1789760774.766060:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.766066:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.766078:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26730496, 26734592) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.766117:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26730496=0x197e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.766187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.766190:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26734592 00000020:00010000:0.0:1789760774.766199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.766202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.766205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.766209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.766212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.766222:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.766228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.766258:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760774.766373:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 26730496<>26734592 00000100:00100000:1.0:1789760774.767215:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.767248:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820288:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760774.812763:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760774.812779:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.812789:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.812870:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760774.812901:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820288:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.812971:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6526, 6526] 00000080:00200000:0.0:1789760774.812988:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.812994:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.813008:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26734592 i_blocks 53255 00000020:00200000:0.0:1789760774.813011:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.813022:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.813198:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26734592, count: 4096 00000080:00200000:0.0:1789760774.813208:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26734592, bytes: 4096 00000080:00200000:0.0:1789760774.813218:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.813226:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.813229:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26734592, 26738688) 0 0 00000080:00200000:0.0:1789760774.813233:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.813249:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6527, 6527] 00020000:00200000:0.0:1789760774.813257:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26734592 chunk: [26734592, 26738688) 26738688, 4096 00020000:00200000:0.0:1789760774.813262:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.813280:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6527, 6527] 00000008:00010000:0.0:1789760774.813298:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.813304:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.813315:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26734592, 26738688) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.813329:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26734592=0x197f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.813385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.813388:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26738688 00000020:00010000:0.0:1789760774.813527:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.813532:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.813535:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.813538:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.813541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.813550:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.813556:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.813573:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760774.813677:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 26734592<>26738688 00000100:00100000:1.0:1789760774.813743:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820416:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.844520:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973846 gen 3 00000008:00010000:1.0:1789760774.844642:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760774.844651:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.844661:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.844682:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760774.844711:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820416:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.845174:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6527, 6527] 00000080:00200000:0.0:1789760774.845190:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.845196:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.845213:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26738688 i_blocks 53255 00000020:00200000:0.0:1789760774.845216:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.845227:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.845260:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26738688, count: 4096 00000080:00200000:0.0:1789760774.845267:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26738688, bytes: 4096 00000080:00200000:0.0:1789760774.845278:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.845286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.845288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26738688, 26742784) 0 0 00000080:00200000:0.0:1789760774.845292:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.845306:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6528, 6528] 00020000:00200000:0.0:1789760774.845315:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26738688 chunk: [26738688, 26742784) 26742784, 4096 00020000:00200000:0.0:1789760774.845319:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.845336:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6528, 6528] 00000008:00010000:0.0:1789760774.845353:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.845359:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.845369:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26738688, 26742784) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.845382:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26738688=0x1980000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.845468:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.845471:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26742784 00000020:00010000:0.0:1789760774.845479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.845483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.845486:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.845489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.845492:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.845499:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.845505:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.845520:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760774.845636:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 26738688<>26742784 00000100:00100000:1.0:1789760774.845807:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.845840:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820544:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760774.877078:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760774.877104:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.877116:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.877146:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760774.877177:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820544:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.877276:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6528, 6528] 00000080:00200000:0.0:1789760774.877292:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.877298:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.877313:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26742784 i_blocks 53255 00000020:00200000:0.0:1789760774.877317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.877327:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.877361:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26742784, count: 4096 00000080:00200000:0.0:1789760774.877368:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26742784, bytes: 4096 00000080:00200000:0.0:1789760774.877377:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.877386:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.877388:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26742784, 26746880) 0 0 00000080:00200000:0.0:1789760774.877421:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.877437:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6529, 6529] 00020000:00200000:0.0:1789760774.877445:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26742784 chunk: [26742784, 26746880) 26746880, 4096 00020000:00200000:0.0:1789760774.877449:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.877466:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6529, 6529] 00000008:00010000:0.0:1789760774.877483:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.877489:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.877501:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26742784, 26746880) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.877515:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26742784=0x1981000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.877649:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.877653:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26746880 00000020:00010000:0.0:1789760774.877662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.877665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.877669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.877672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.877675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.877685:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.877691:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.877707:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760774.877805:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 26742784<>26746880 00000100:00100000:1.0:1789760774.878041:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820672:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.915783:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973847 gen 3 00000100:00100000:1.0:1789760774.915824:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ed80 x1876699493820672/t4294973848(4294973848) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760790 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760774.915874:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760774.915882:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.915891:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.915914:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760774.915943:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820672:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.916128:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6529, 6529] 00000080:00200000:0.0:1789760774.916155:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.916162:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.916176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26746880 i_blocks 53255 00000020:00200000:0.0:1789760774.916181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.916192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.916229:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26746880, count: 4096 00000080:00200000:0.0:1789760774.916236:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26746880, bytes: 4096 00000080:00200000:0.0:1789760774.916246:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.916254:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.916256:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26746880, 26750976) 0 0 00000080:00200000:0.0:1789760774.916261:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.916277:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6530, 6530] 00020000:00200000:0.0:1789760774.916286:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26746880 chunk: [26746880, 26750976) 26750976, 4096 00020000:00200000:0.0:1789760774.916291:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.916313:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6530, 6530] 00000008:00010000:0.0:1789760774.916333:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.916339:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.916352:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26746880, 26750976) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.916366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26746880=0x1982000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.916445:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.916448:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26750976 00000020:00010000:0.0:1789760774.916455:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.916459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.916462:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.916466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.916469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.916477:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.916483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.916498:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760774.916717:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 26746880<>26750976 00000100:00100000:1.0:1789760774.916833:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493820800:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.948530:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973849 gen 3 00000008:00010000:1.0:1789760774.948610:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760774.948619:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.948629:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.948653:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760774.948683:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493820800:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.948846:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6530, 6530] 00000080:00200000:0.0:1789760774.948863:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.948869:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.948886:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26750976 i_blocks 53255 00000020:00200000:0.0:1789760774.948890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.948900:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.948936:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26750976, count: 4096 00000080:00200000:0.0:1789760774.948945:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26750976, bytes: 4096 00000080:00200000:0.0:1789760774.948955:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.948965:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.948967:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26750976, 26755072) 0 0 00000080:00200000:0.0:1789760774.948972:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.948987:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6531, 6531] 00020000:00200000:0.0:1789760774.948997:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26750976 chunk: [26750976, 26755072) 26755072, 4096 00020000:00200000:0.0:1789760774.949001:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.949021:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6531, 6531] 00000008:00010000:0.0:1789760774.949040:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.949046:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.949058:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26750976, 26755072) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.949073:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26750976=0x1983000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.949152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.949155:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26755072 00000020:00010000:0.0:1789760774.949165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.949168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.949171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.949175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.949178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.949187:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.949193:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.949209:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760774.949310:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 26750976<>26755072 00000100:00100000:1.0:1789760774.949513:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760774.980036:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973850 gen 3 00000008:00010000:1.0:1789760774.980098:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760774.980107:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.980117:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.980140:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760774.980171:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493820928:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760774.980498:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6531, 6531] 00000080:00200000:0.0:1789760774.980515:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.980521:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.980537:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26755072 i_blocks 53255 00000020:00200000:0.0:1789760774.980541:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.980552:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.980595:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26755072, count: 4096 00000080:00200000:0.0:1789760774.980603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26755072, bytes: 4096 00000080:00200000:0.0:1789760774.980613:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.980621:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.980624:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26755072, 26759168) 0 0 00000080:00200000:0.0:1789760774.980629:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.980645:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6532, 6532] 00020000:00200000:0.0:1789760774.980655:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26755072 chunk: [26755072, 26759168) 26759168, 4096 00020000:00200000:0.0:1789760774.980660:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.980680:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6532, 6532] 00000008:00010000:0.0:1789760774.980699:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.980706:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760774.980717:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26755072, 26759168) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.980732:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26755072=0x1984000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.980789:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.980792:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26759168 00000020:00010000:0.0:1789760774.980800:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.980804:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.980807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.980811:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.980813:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.980833:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.980840:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.980857:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760774.980954:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 26755072<>26759168 00000100:00100000:1.0:1789760774.981185:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493821056:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760775.011911:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760775.011928:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.011941:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.011964:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760775.011994:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493821056:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.012427:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6532, 6532] 00000080:00200000:0.0:1789760775.012444:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.012450:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.012466:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26759168 i_blocks 53255 00000020:00200000:0.0:1789760775.012471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.012482:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.012516:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26759168, count: 4096 00000080:00200000:0.0:1789760775.012523:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26759168, bytes: 4096 00000080:00200000:0.0:1789760775.012535:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.012544:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.012546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26759168, 26763264) 0 0 00000080:00200000:0.0:1789760775.012552:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.012643:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6533, 6533] 00020000:00200000:0.0:1789760775.012653:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26759168 chunk: [26759168, 26763264) 26763264, 4096 00020000:00200000:0.0:1789760775.012658:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.012680:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6533, 6533] 00000008:00010000:0.0:1789760775.012698:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.012705:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.012718:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26759168, 26763264) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.012734:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26759168=0x1985000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.012796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.012799:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26763264 00000020:00010000:0.0:1789760775.012809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.012812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.012826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.012830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.012832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.012842:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.012849:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.012867:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760775.012973:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 26759168<>26763264 00000100:00100000:1.0:1789760775.013045:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493821184:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760775.039996:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973852 gen 3 00000008:00010000:1.0:1789760775.040064:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760775.040073:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.040094:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.040259:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760775.040292:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493821184:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.040371:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6533, 6533] 00000080:00200000:0.0:1789760775.040388:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.040421:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.040437:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26763264 i_blocks 53255 00000020:00200000:0.0:1789760775.040441:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.040451:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.040485:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26763264, count: 4096 00000080:00200000:0.0:1789760775.040493:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26763264, bytes: 4096 00000080:00200000:0.0:1789760775.040504:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.040513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.040515:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26763264, 26767360) 0 0 00000080:00200000:0.0:1789760775.040520:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.040537:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6534, 6534] 00020000:00200000:0.0:1789760775.040546:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26763264 chunk: [26763264, 26767360) 26767360, 4096 00020000:00200000:0.0:1789760775.040550:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.040578:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6534, 6534] 00000008:00010000:0.0:1789760775.040596:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.040602:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.040614:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26763264, 26767360) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.040628:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26763264=0x1986000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.040685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.040687:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26767360 00000020:00010000:0.0:1789760775.040697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.040700:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.040704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.040708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.040710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.040719:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.040726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.040743:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760775.041126:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 26763264<>26767360 00000100:00100000:1.0:1789760775.041279:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493821312:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760775.080304:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760775.080321:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.080334:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.080440:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760775.080471:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493821312:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760775.080618:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6534, 6534] 00000080:00200000:3.0:1789760775.080635:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.080640:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.080657:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26767360 i_blocks 53255 00000020:00200000:3.0:1789760775.080661:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.080738:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.080772:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26767360, count: 4096 00000080:00200000:3.0:1789760775.080779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26767360, bytes: 4096 00000080:00200000:3.0:1789760775.080789:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.080799:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.080802:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26767360, 26771456) 0 0 00000080:00200000:3.0:1789760775.080806:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.080885:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6535, 6535] 00020000:00200000:3.0:1789760775.080895:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26767360 chunk: [26767360, 26771456) 26771456, 4096 00020000:00200000:3.0:1789760775.080900:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.080919:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6535, 6535] 00000008:00010000:3.0:1789760775.080987:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.080994:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760775.081007:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26767360, 26771456) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.081021:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26767360=0x1987000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.081090:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.081093:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26771456 00000020:00010000:3.0:1789760775.081102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.081106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.081109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.081113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.081116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.081124:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.081131:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.081148:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760775.081252:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 26767360<>26771456 00000100:00100000:2.0:1789760775.081331:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821440:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.116130:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973854 gen 3 00000008:00010000:3.0:1789760775.116200:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760775.116209:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.116219:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.116244:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000080:00200000:0.0:1789760775.116278:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6535, 6535] 00000080:00200000:0.0:1789760775.116298:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.116303:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.116318:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26771456 i_blocks 53255 00000020:00200000:0.0:1789760775.116322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.116332:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760775.116346:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821440:192.168.203.160@tcp:4:dd.0 00000080:00200008:0.0:1789760775.116367:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26771456, count: 4096 00000080:00200000:0.0:1789760775.116374:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26771456, bytes: 4096 00000080:00200000:0.0:1789760775.116384:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.116423:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.116425:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26771456, 26775552) 0 0 00000080:00200000:0.0:1789760775.116430:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.116488:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6536, 6536] 00020000:00200000:0.0:1789760775.116496:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26771456 chunk: [26771456, 26775552) 26775552, 4096 00020000:00200000:0.0:1789760775.116499:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.116516:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6536, 6536] 00000008:00010000:0.0:1789760775.116530:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.116535:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.116544:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26771456, 26775552) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.116555:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26771456=0x1988000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.116609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.116611:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26775552 00000020:00010000:0.0:1789760775.116618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.116620:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.116623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.116625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.116627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.116633:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.116638:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.116648:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760775.116732:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26771456<>26775552 00000100:00100000:0.0:1789760775.116770:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493821568:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760775.157980:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973855 gen 3 00000008:00010000:0.0:1789760775.158030:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760775.158039:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.158049:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.158140:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760775.158170:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493821568:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760775.159216:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6536, 6536] 00000080:00200000:3.0:1789760775.159234:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.159241:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.159258:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26775552 i_blocks 53255 00000020:00200000:3.0:1789760775.159263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.159277:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.159469:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26775552, count: 4096 00000080:00200000:3.0:1789760775.159478:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26775552, bytes: 4096 00000080:00200000:3.0:1789760775.159493:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.159502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.159504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26775552, 26779648) 0 0 00000080:00200000:3.0:1789760775.160702:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.160727:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6537, 6537] 00020000:00200000:3.0:1789760775.160803:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26775552 chunk: [26775552, 26779648) 26779648, 4096 00020000:00200000:3.0:1789760775.160809:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.160833:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6537, 6537] 00000008:00010000:3.0:1789760775.160853:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.160859:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760775.160872:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26775552, 26779648) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.160886:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26775552=0x1989000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.160949:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.160952:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26779648 00000020:00010000:3.0:1789760775.160960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.160963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.160978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.160982:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.160985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.160996:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.161003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.161029:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760775.161380:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 26775552<>26779648 00000100:00100000:2.0:1789760775.161760:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.161912:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.218982:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973856 gen 3 00000008:00010000:3.0:1789760775.219035:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760775.219045:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.219056:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.219089:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760775.219121:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.219349:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6537, 6537] 00000080:00200000:2.0:1789760775.219366:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.219372:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.219388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26779648 i_blocks 53255 00000020:00200000:2.0:1789760775.219423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.219434:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.219471:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26779648, count: 4096 00000080:00200000:2.0:1789760775.219479:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26779648, bytes: 4096 00000080:00200000:2.0:1789760775.219489:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.219497:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.219499:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26779648, 26783744) 0 0 00000080:00200000:2.0:1789760775.219504:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.219521:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6538, 6538] 00020000:00200000:2.0:1789760775.219529:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26779648 chunk: [26779648, 26783744) 26783744, 4096 00020000:00200000:2.0:1789760775.219533:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.219554:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6538, 6538] 00000008:00010000:2.0:1789760775.219631:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.219638:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.219651:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26779648, 26783744) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.219665:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26779648=0x198a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.219719:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.219722:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26783744 00000020:00010000:2.0:1789760775.219731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.219734:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.219738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.219741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.219744:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.219751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.219758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.219773:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760775.219885:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26779648<>26783744 00000100:00100000:3.0:1789760775.219948:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.254468:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973857 gen 3 00000008:00010000:3.0:1789760775.254523:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760775.254532:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.254543:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.254675:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760775.254706:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493821824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.255321:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6538, 6538] 00000080:00200000:2.0:1789760775.255342:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.255348:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.255365:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26783744 i_blocks 53255 00000020:00200000:2.0:1789760775.255369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.255381:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.255450:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26783744, count: 4096 00000080:00200000:2.0:1789760775.255458:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26783744, bytes: 4096 00000080:00200000:2.0:1789760775.255470:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.255479:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.255482:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26783744, 26787840) 0 0 00000080:00200000:2.0:1789760775.255487:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.255504:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6539, 6539] 00020000:00200000:2.0:1789760775.255513:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26783744 chunk: [26783744, 26787840) 26787840, 4096 00020000:00200000:2.0:1789760775.255517:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.255537:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6539, 6539] 00000008:00010000:2.0:1789760775.255556:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.255633:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.255712:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26783744, 26787840) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.255728:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26783744=0x198b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.255788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.255791:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26787840 00000020:00010000:2.0:1789760775.255799:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.255803:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.255806:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.255809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.255812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.255874:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.255881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.255899:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760775.256003:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26783744<>26787840 00000100:00100000:3.0:1789760775.256231:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493821952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.314694:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973858 gen 3 00000008:00010000:3.0:1789760775.314752:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760775.314761:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.314775:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.314794:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000080:00200000:2.0:1789760775.314900:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6539, 6539] 00000080:00200000:2.0:1789760775.314919:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.314925:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.314940:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26787840 i_blocks 53255 00000020:00200000:2.0:1789760775.314944:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.314955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.314990:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26787840, count: 4096 00000080:00200000:2.0:1789760775.315000:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26787840, bytes: 4096 00000080:00200000:2.0:1789760775.315010:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.315019:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.315022:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26787840, 26791936) 0 0 00000080:00200000:2.0:1789760775.315027:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.315044:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6540, 6540] 00020000:00200000:2.0:1789760775.315053:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26787840 chunk: [26787840, 26791936) 26791936, 4096 00020000:00200000:2.0:1789760775.315057:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.315089:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6540, 6540] 00000008:00010000:2.0:1789760775.315109:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.315116:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.315128:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26787840, 26791936) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.315209:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26787840=0x198c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.315261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.315264:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26791936 00000020:00010000:2.0:1789760775.315273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.315277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.315280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.315284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.315287:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.315294:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.315301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760775.315357:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493821952:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760775.315445:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:3.0:1789760775.315544:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 26787840<>26791936 00000100:00100000:3.0:1789760775.315653:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.315668:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.358636:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973859 gen 3 00000008:00010000:3.0:1789760775.358689:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760775.358698:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.358710:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.358733:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760775.358762:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.358965:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6540, 6540] 00000080:00200000:2.0:1789760775.358982:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.358988:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.359004:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26791936 i_blocks 53255 00000020:00200000:2.0:1789760775.359007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.359018:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.359063:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26791936, count: 4096 00000080:00200000:2.0:1789760775.359071:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26791936, bytes: 4096 00000080:00200000:2.0:1789760775.359091:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.359100:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.359103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26791936, 26796032) 0 0 00000080:00200000:2.0:1789760775.359108:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.359123:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6541, 6541] 00020000:00200000:2.0:1789760775.359141:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26791936 chunk: [26791936, 26796032) 26796032, 4096 00020000:00200000:2.0:1789760775.359146:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.359167:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6541, 6541] 00000008:00010000:2.0:1789760775.359186:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.359191:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.359202:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26791936, 26796032) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.359217:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26791936=0x198d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.359280:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.359283:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26796032 00000020:00010000:2.0:1789760775.359292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.359296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.359299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.359302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.359305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.359313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.359320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.359337:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760775.359478:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26791936<>26796032 00000100:00100000:2.0:1789760775.359517:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.359535:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493822208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.393788:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973860 gen 3 00000008:00010000:3.0:1789760775.393853:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760775.393861:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.393871:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.393891:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760775.393918:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493822208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.394017:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6541, 6541] 00000080:00200000:2.0:1789760775.394034:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.394040:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.394055:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26796032 i_blocks 53255 00000020:00200000:2.0:1789760775.394059:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.394070:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.394114:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26796032, count: 4096 00000080:00200000:2.0:1789760775.394121:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26796032, bytes: 4096 00000080:00200000:2.0:1789760775.394140:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.394150:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.394152:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26796032, 26800128) 0 0 00000080:00200000:2.0:1789760775.394158:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.394175:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6542, 6542] 00020000:00200000:2.0:1789760775.394184:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26796032 chunk: [26796032, 26800128) 26800128, 4096 00020000:00200000:2.0:1789760775.394189:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.394208:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6542, 6542] 00000008:00010000:2.0:1789760775.394228:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.394234:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.394246:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26796032, 26800128) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.394260:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26796032=0x198e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.394318:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.394321:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26800128 00000020:00010000:2.0:1789760775.394329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.394333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.394336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.394340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.394343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.394352:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.394359:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.394376:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760775.394739:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26796032<>26800128 00000100:00100000:3.0:1789760775.394800:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493822336:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760775.408268:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760775.429888:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973861 gen 3 00000008:00010000:3.0:1789760775.429946:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760775.429955:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.429968:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.429992:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760775.430025:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493822336:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.430152:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6542, 6542] 00000080:00200000:0.0:1789760775.430169:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.430176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.430192:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26800128 i_blocks 53255 00000020:00200000:0.0:1789760775.430196:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.430207:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.430242:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26800128, count: 4096 00000080:00200000:0.0:1789760775.430249:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26800128, bytes: 4096 00000080:00200000:0.0:1789760775.430259:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.430268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.430270:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26800128, 26804224) 0 0 00000080:00200000:0.0:1789760775.430275:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.430292:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6543, 6543] 00020000:00200000:0.0:1789760775.430302:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26800128 chunk: [26800128, 26804224) 26804224, 4096 00020000:00200000:0.0:1789760775.430306:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.430326:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6543, 6543] 00000008:00010000:0.0:1789760775.430344:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.430350:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.430361:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26800128, 26804224) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.430375:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26800128=0x198f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.430464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.430467:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26804224 00000020:00010000:0.0:1789760775.430475:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.430479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.430482:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.430486:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.430489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.430497:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.430505:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.430522:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760775.430641:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26800128<>26804224 00000100:00100000:0.0:1789760775.430691:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.430706:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493822464:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760775.463532:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973862 gen 3 00000008:00010000:0.0:1789760775.463596:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760775.463604:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.463615:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.463637:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000080:00200000:3.0:1789760775.463768:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6543, 6543] 00000080:00200000:3.0:1789760775.463786:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.463792:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.463809:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26804224 i_blocks 53255 00000020:00200000:3.0:1789760775.463814:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.463834:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.463870:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26804224, count: 4096 00000080:00200000:3.0:1789760775.463879:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26804224, bytes: 4096 00000080:00200000:3.0:1789760775.463890:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.463901:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.463903:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26804224, 26808320) 0 0 00000080:00200000:3.0:1789760775.463908:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.463926:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6544, 6544] 00020000:00200000:3.0:1789760775.463935:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26804224 chunk: [26804224, 26808320) 26808320, 4096 00020000:00200000:3.0:1789760775.463939:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.463960:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6544, 6544] 00000008:00010000:3.0:1789760775.463979:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.463986:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760775.463999:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26804224, 26808320) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.464014:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26804224=0x1990000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.464095:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.464098:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26808320 00000020:00010000:3.0:1789760775.464106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.464110:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.464114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.464118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.464121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.464239:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.464249:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.464268:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760775.464369:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 26804224<>26808320 00000100:00100000:2.0:1789760775.464479:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822592:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760775.465235:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493822464:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760775.494111:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760775.494127:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.494139:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.494221:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760775.494254:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822592:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.494759:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6544, 6544] 00000080:00200000:0.0:1789760775.494778:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.494786:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.494803:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26808320 i_blocks 53255 00000020:00200000:0.0:1789760775.494807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.494877:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.494915:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26808320, count: 4096 00000080:00200000:0.0:1789760775.494924:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26808320, bytes: 4096 00000080:00200000:0.0:1789760775.494936:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.494945:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.494947:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26808320, 26812416) 0 0 00000080:00200000:0.0:1789760775.494952:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.494969:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6545, 6545] 00020000:00200000:0.0:1789760775.494978:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26808320 chunk: [26808320, 26812416) 26812416, 4096 00020000:00200000:0.0:1789760775.494983:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.495004:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6545, 6545] 00000008:00010000:0.0:1789760775.495024:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.495030:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.495043:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26808320, 26812416) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.495059:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26808320=0x1991000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.495303:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.495306:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26812416 00000020:00010000:0.0:1789760775.495317:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.495320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.495324:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.495327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.495330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.495339:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.495346:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.495369:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760775.495635:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 26808320<>26812416 00000100:00100000:1.0:1789760775.495702:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760775.496036:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493822720:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760775.530896:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973863 gen 3 00000100:00100000:1.0:1789760775.530928:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304f800 x1876699493822720/t4294973864(4294973864) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760791 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760775.530967:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760775.530973:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.530981:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.531005:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760775.531031:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493822720:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760775.531261:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6545, 6545] 00000080:00200000:3.0:1789760775.531274:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760775.531279:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760775.531290:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26812416 i_blocks 53255 00000020:00200000:3.0:1789760775.531294:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760775.531304:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760775.531332:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26812416, count: 4096 00000080:00200000:3.0:1789760775.531339:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26812416, bytes: 4096 00000080:00200000:3.0:1789760775.531348:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760775.531355:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760775.531357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26812416, 26816512) 0 0 00000080:00200000:3.0:1789760775.531361:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760775.531373:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6546, 6546] 00020000:00200000:3.0:1789760775.531379:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26812416 chunk: [26812416, 26816512) 26816512, 4096 00020000:00200000:3.0:1789760775.531382:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760775.531434:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6546, 6546] 00000008:00010000:3.0:1789760775.531449:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760775.531454:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760775.531463:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26812416, 26816512) append 0 append lockpos 0 00000080:00200000:3.0:1789760775.531473:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26812416=0x1992000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760775.531518:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760775.531521:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26816512 00000020:00010000:3.0:1789760775.531528:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760775.531532:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.531534:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760775.531537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760775.531540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760775.531546:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760775.531551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760775.531787:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760775.531869:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 26812416<>26816512 00000100:00100000:3.0:1789760775.531910:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.531924:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.566903:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973865 gen 3 00000008:00010000:3.0:1789760775.566951:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760775.566957:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.566965:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.567048:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760775.567073:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493822848:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.567157:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6546, 6546] 00000080:00200000:0.0:1789760775.567170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.567175:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.567186:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26816512 i_blocks 53255 00000020:00200000:0.0:1789760775.567189:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.567198:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.567226:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26816512, count: 4096 00000080:00200000:0.0:1789760775.567232:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26816512, bytes: 4096 00000080:00200000:0.0:1789760775.567240:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.567247:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.567250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26816512, 26820608) 0 0 00000080:00200000:0.0:1789760775.567253:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.567266:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6547, 6547] 00020000:00200000:0.0:1789760775.567272:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26816512 chunk: [26816512, 26820608) 26820608, 4096 00020000:00200000:0.0:1789760775.567276:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.567291:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6547, 6547] 00000008:00010000:0.0:1789760775.567305:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.567309:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.567320:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26816512, 26820608) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.567331:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26816512=0x1993000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.567742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.567746:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26820608 00000020:00010000:0.0:1789760775.567753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.567757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.567759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.567762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.567765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.567771:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.567777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.567792:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760775.567876:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 26816512<>26820608 00000100:00100000:1.0:1789760775.567968:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493822976:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760775.606448:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760775.606458:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.606467:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.606486:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760775.606512:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493822976:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760775.607020:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6547, 6547] 00000080:00200000:0.0:1789760775.607032:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.607037:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.607048:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26820608 i_blocks 53255 00000020:00200000:0.0:1789760775.607051:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.607060:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.607096:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26820608, count: 4096 00000080:00200000:0.0:1789760775.607102:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26820608, bytes: 4096 00000080:00200000:0.0:1789760775.607110:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.607117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.607118:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26820608, 26824704) 0 0 00000080:00200000:0.0:1789760775.607122:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.607134:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6548, 6548] 00020000:00200000:0.0:1789760775.607141:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26820608 chunk: [26820608, 26824704) 26824704, 4096 00020000:00200000:0.0:1789760775.607145:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.607159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6548, 6548] 00000008:00010000:0.0:1789760775.607172:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.607176:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760775.607185:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26820608, 26824704) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.607196:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26820608=0x1994000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.607237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.607239:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26824704 00000020:00010000:0.0:1789760775.607245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.607248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.607251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.607255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.607258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.607263:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.607268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.607283:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760775.607356:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 26820608<>26824704 00000100:00100000:1.0:1789760775.608058:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493823104:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760775.670920:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973867 gen 3 00000008:00010000:1.0:1789760775.670966:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760775.670972:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.670980:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.671004:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760775.671032:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493823104:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.671133:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6548, 6548] 00000080:00200000:2.0:1789760775.671146:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.671151:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.671164:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26824704 i_blocks 53255 00000020:00200000:2.0:1789760775.671167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.671176:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.671203:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26824704, count: 4096 00000080:00200000:2.0:1789760775.671210:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26824704, bytes: 4096 00000080:00200000:2.0:1789760775.671220:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.671227:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.671230:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26824704, 26828800) 0 0 00000080:00200000:2.0:1789760775.671233:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.671248:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6549, 6549] 00020000:00200000:2.0:1789760775.671255:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26824704 chunk: [26824704, 26828800) 26828800, 4096 00020000:00200000:2.0:1789760775.671258:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.671272:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6549, 6549] 00000008:00010000:2.0:1789760775.671287:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.671291:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.671301:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26824704, 26828800) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.671312:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26824704=0x1995000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.671358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.671361:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26828800 00000020:00010000:2.0:1789760775.671616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.671621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.671624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.671627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.671629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.671636:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.671641:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.671655:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760775.671741:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26824704<>26828800 00000100:00100000:2.0:1789760775.671780:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760775.671793:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.707236:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973868 gen 3 00000008:00010000:3.0:1789760775.707272:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760775.707280:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.707289:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.707360:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760775.707433:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823232:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.707765:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6549, 6549] 00000080:00200000:2.0:1789760775.707778:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.707783:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.707795:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26828800 i_blocks 53255 00000020:00200000:2.0:1789760775.707868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.707880:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.707908:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26828800, count: 4096 00000080:00200000:2.0:1789760775.707914:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26828800, bytes: 4096 00000080:00200000:2.0:1789760775.707923:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.707930:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.707932:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26828800, 26832896) 0 0 00000080:00200000:2.0:1789760775.707937:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.707949:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6550, 6550] 00020000:00200000:2.0:1789760775.707956:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26828800 chunk: [26828800, 26832896) 26832896, 4096 00020000:00200000:2.0:1789760775.707959:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.707973:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6550, 6550] 00000008:00010000:2.0:1789760775.707987:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.708063:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.708084:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26828800, 26832896) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.708095:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26828800=0x1996000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.708141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.708143:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26832896 00000020:00010000:2.0:1789760775.708151:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.708153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.708156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.708159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.708161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.708168:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.708173:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.708188:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760775.708267:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26828800<>26832896 00000100:00100000:3.0:1789760775.708325:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823360:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.749619:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973869 gen 3 00000008:00010000:3.0:1789760775.749664:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760775.749670:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.749679:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.749749:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760775.749774:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.751508:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6550, 6550] 00000080:00200000:2.0:1789760775.751528:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.751534:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.751547:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26832896 i_blocks 53255 00000020:00200000:2.0:1789760775.751551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.751563:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.751591:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26832896, count: 4096 00000080:00200000:2.0:1789760775.751597:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26832896, bytes: 4096 00000080:00200000:2.0:1789760775.751608:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.751615:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.751617:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26832896, 26836992) 0 0 00000080:00200000:2.0:1789760775.751621:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.751635:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6551, 6551] 00020000:00200000:2.0:1789760775.751641:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26832896 chunk: [26832896, 26836992) 26836992, 4096 00020000:00200000:2.0:1789760775.751645:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.751661:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6551, 6551] 00000008:00010000:2.0:1789760775.751674:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.751678:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.751686:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26832896, 26836992) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.751696:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26832896=0x1997000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.752583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.752589:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26836992 00000020:00010000:2.0:1789760775.752599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.752603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.752606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.752609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.752611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.752619:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.752625:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.752664:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760775.752755:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26832896<>26836992 00000100:00100000:3.0:1789760775.752845:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.804746:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973870 gen 3 00000008:00010000:3.0:1789760775.804782:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760775.804789:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.804797:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.804927:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760775.804964:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823488:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.805324:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6551, 6551] 00000080:00200000:2.0:1789760775.805340:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.805345:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.805357:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26836992 i_blocks 53255 00000020:00200000:2.0:1789760775.805360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.805369:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.805531:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26836992, count: 4096 00000080:00200000:2.0:1789760775.805539:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26836992, bytes: 4096 00000080:00200000:2.0:1789760775.805549:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.805565:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.805567:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26836992, 26841088) 0 0 00000080:00200000:2.0:1789760775.805571:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.805582:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6552, 6552] 00020000:00200000:2.0:1789760775.805589:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26836992 chunk: [26836992, 26841088) 26841088, 4096 00020000:00200000:2.0:1789760775.805593:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.805609:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6552, 6552] 00000008:00010000:2.0:1789760775.805622:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.805627:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.805636:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26836992, 26841088) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.805648:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26836992=0x1998000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.805692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.805695:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26841088 00000020:00010000:2.0:1789760775.805701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.805704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.805707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.805709:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.805711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.805718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.805723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.805738:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760775.806548:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26836992<>26841088 00000100:00100000:3.0:1789760775.806863:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760775.806883:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.846136:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973871 gen 3 00000008:00010000:3.0:1789760775.846172:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760775.846179:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.846188:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.846208:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760775.846232:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823616:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.846429:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6552, 6552] 00000080:00200000:2.0:1789760775.846443:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.846448:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.846460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26841088 i_blocks 53255 00000020:00200000:2.0:1789760775.846463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.846472:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.846498:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26841088, count: 4096 00000080:00200000:2.0:1789760775.846504:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26841088, bytes: 4096 00000080:00200000:2.0:1789760775.846512:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.846519:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.846520:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26841088, 26845184) 0 0 00000080:00200000:2.0:1789760775.846524:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.846537:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6553, 6553] 00020000:00200000:2.0:1789760775.846545:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26841088 chunk: [26841088, 26845184) 26845184, 4096 00020000:00200000:2.0:1789760775.846548:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.846572:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6553, 6553] 00000008:00010000:2.0:1789760775.846586:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.846591:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.846600:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26841088, 26845184) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.846611:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26841088=0x1999000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.846654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.846657:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26845184 00000020:00010000:2.0:1789760775.846664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.846667:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.846669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.846672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.846675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.846682:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.846687:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.846701:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760775.846778:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26841088<>26845184 00000100:00100000:3.0:1789760775.846863:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823744:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760775.889893:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760775.889904:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.889912:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.889932:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760775.889956:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493823744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.890418:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6553, 6553] 00000080:00200000:2.0:1789760775.890431:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.890436:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.890448:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26845184 i_blocks 53255 00000020:00200000:2.0:1789760775.890451:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.890459:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.890485:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26845184, count: 4096 00000080:00200000:2.0:1789760775.890490:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26845184, bytes: 4096 00000080:00200000:2.0:1789760775.890498:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.890504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.890506:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26845184, 26849280) 0 0 00000080:00200000:2.0:1789760775.890510:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.890522:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6554, 6554] 00020000:00200000:2.0:1789760775.890528:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26845184 chunk: [26845184, 26849280) 26849280, 4096 00020000:00200000:2.0:1789760775.890532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.890546:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6554, 6554] 00000008:00010000:2.0:1789760775.890558:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.890563:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.890571:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26845184, 26849280) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.890581:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26845184=0x199a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.890622:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.890625:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26849280 00000020:00010000:2.0:1789760775.890632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.890635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.890638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.890641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.890644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.890651:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.890656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.890670:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760775.890742:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26845184<>26849280 00000100:00100000:3.0:1789760775.891177:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823872:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.925996:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973873 gen 3 00000008:00010000:3.0:1789760775.926058:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760775.926066:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.926085:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.926196:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760775.926223:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493823872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.926292:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6554, 6554] 00000080:00200000:2.0:1789760775.926308:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.926314:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.926329:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26849280 i_blocks 53255 00000020:00200000:2.0:1789760775.926332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.926342:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.926373:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26849280, count: 4096 00000080:00200000:2.0:1789760775.926380:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26849280, bytes: 4096 00000080:00200000:2.0:1789760775.926391:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.926419:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.926421:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26849280, 26853376) 0 0 00000080:00200000:2.0:1789760775.926426:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.926441:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6555, 6555] 00020000:00200000:2.0:1789760775.926449:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26849280 chunk: [26849280, 26853376) 26853376, 4096 00020000:00200000:2.0:1789760775.926454:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.926471:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6555, 6555] 00000008:00010000:2.0:1789760775.926487:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.926493:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.926504:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26849280, 26853376) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.926516:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26849280=0x199b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.926631:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.926634:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26853376 00000020:00010000:2.0:1789760775.926642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.926645:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.926649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.926652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.926655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.926662:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.926667:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.926681:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760775.926774:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26849280<>26853376 00000100:00100000:3.0:1789760775.926838:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760775.967049:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973874 gen 3 00000008:00010000:3.0:1789760775.967103:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760775.967112:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.967124:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.967241:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760775.967274:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760775.967362:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6555, 6555] 00000080:00200000:2.0:1789760775.967379:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.967385:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.967426:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26853376 i_blocks 53255 00000020:00200000:2.0:1789760775.967430:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.967441:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.967475:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26853376, count: 4096 00000080:00200000:2.0:1789760775.967482:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26853376, bytes: 4096 00000080:00200000:2.0:1789760775.967491:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.967500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.967503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26853376, 26857472) 0 0 00000080:00200000:2.0:1789760775.967508:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.967524:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6556, 6556] 00020000:00200000:2.0:1789760775.967533:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26853376 chunk: [26853376, 26857472) 26857472, 4096 00020000:00200000:2.0:1789760775.967537:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.967626:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6556, 6556] 00000008:00010000:2.0:1789760775.967646:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.967652:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760775.967665:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26853376, 26857472) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.967679:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26853376=0x199c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.967737:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.967740:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26857472 00000020:00010000:2.0:1789760775.967748:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.967752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.967755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.967759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.967762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.967769:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.967776:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.967792:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760775.967967:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 26853376<>26857472 00000100:00100000:3.0:1789760775.968022:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.000459:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973875 gen 3 00000008:00010000:3.0:1789760776.000516:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760776.000525:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.000537:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.000599:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760776.000629:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824128:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.000931:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6556, 6556] 00000080:00200000:2.0:1789760776.000951:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.000957:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.000974:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26857472 i_blocks 53255 00000020:00200000:2.0:1789760776.000978:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.000990:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.001029:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26857472, count: 4096 00000080:00200000:2.0:1789760776.001037:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26857472, bytes: 4096 00000080:00200000:2.0:1789760776.001047:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.001056:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.001059:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26857472, 26861568) 0 0 00000080:00200000:2.0:1789760776.001064:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.001331:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6557, 6557] 00020000:00200000:2.0:1789760776.001340:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26857472 chunk: [26857472, 26861568) 26861568, 4096 00020000:00200000:2.0:1789760776.001344:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.001369:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6557, 6557] 00000008:00010000:2.0:1789760776.001389:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.001483:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.001497:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26857472, 26861568) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.001515:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26857472=0x199d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.001573:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.001577:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26861568 00000020:00010000:2.0:1789760776.001585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.001588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.001591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.001595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.001598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.001607:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.001615:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.001635:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760776.001746:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 26857472<>26861568 00000100:00100000:3.0:1789760776.001824:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824256:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.047433:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973876 gen 3 00000008:00010000:3.0:1789760776.047492:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760776.047501:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.047513:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.047535:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760776.047563:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.047733:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6557, 6557] 00000080:00200000:2.0:1789760776.047749:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.047755:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.047772:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26861568 i_blocks 53255 00000020:00200000:2.0:1789760776.047777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.047788:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.047833:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26861568, count: 4096 00000080:00200000:2.0:1789760776.047841:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26861568, bytes: 4096 00000080:00200000:2.0:1789760776.047851:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.047859:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.047861:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26861568, 26865664) 0 0 00000080:00200000:2.0:1789760776.047866:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.047883:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6558, 6558] 00020000:00200000:2.0:1789760776.047892:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26861568 chunk: [26861568, 26865664) 26865664, 4096 00020000:00200000:2.0:1789760776.047898:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.047918:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6558, 6558] 00000008:00010000:2.0:1789760776.047937:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.047943:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.047954:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26861568, 26865664) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.047968:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26861568=0x199e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.048024:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.048027:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26865664 00000020:00010000:2.0:1789760776.048034:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.048038:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.048041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.048044:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.048047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.048055:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.048062:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.048296:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760776.048445:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 26861568<>26865664 00000100:00100000:3.0:1789760776.048686:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.091312:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973877 gen 3 00000008:00010000:3.0:1789760776.091368:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760776.091377:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.091389:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.091495:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760776.091525:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.091614:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6558, 6558] 00000080:00200000:2.0:1789760776.091633:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.091640:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.091656:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26865664 i_blocks 53255 00000020:00200000:2.0:1789760776.091660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.091671:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.091706:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26865664, count: 4096 00000080:00200000:2.0:1789760776.091713:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26865664, bytes: 4096 00000080:00200000:2.0:1789760776.091727:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.091736:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.091739:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26865664, 26869760) 0 0 00000080:00200000:2.0:1789760776.091743:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.091899:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6559, 6559] 00020000:00200000:2.0:1789760776.091909:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26865664 chunk: [26865664, 26869760) 26869760, 4096 00020000:00200000:2.0:1789760776.091915:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.091937:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6559, 6559] 00000008:00010000:2.0:1789760776.092038:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.092045:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.092060:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26865664, 26869760) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.092085:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26865664=0x199f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.092147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.092150:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26869760 00000020:00010000:2.0:1789760776.092231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.092236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.092239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.092244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.092247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.092256:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.092265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.092283:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760776.092389:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 26865664<>26869760 00000100:00100000:3.0:1789760776.092502:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.129614:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973878 gen 3 00000008:00010000:3.0:1789760776.129664:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760776.129671:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.129682:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.129698:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760776.129720:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493824512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.129860:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6559, 6559] 00000080:00200000:2.0:1789760776.129879:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.129885:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.129901:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26869760 i_blocks 53255 00000020:00200000:2.0:1789760776.129904:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.129914:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.130159:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26869760, count: 4096 00000080:00200000:2.0:1789760776.130292:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26869760, bytes: 4096 00000080:00200000:2.0:1789760776.130303:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.130312:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.130314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26869760, 26873856) 0 0 00000080:00200000:2.0:1789760776.130319:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.130334:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6560, 6560] 00020000:00200000:2.0:1789760776.130342:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26869760 chunk: [26869760, 26873856) 26873856, 4096 00020000:00200000:2.0:1789760776.130346:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.130365:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6560, 6560] 00000008:00010000:2.0:1789760776.130381:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.130387:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.130420:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26869760, 26873856) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.130433:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26869760=0x19a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.130479:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.130482:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26873856 00000020:00010000:2.0:1789760776.130489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.130493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.130496:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.130499:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.130502:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.130508:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.130515:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.130527:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760776.130632:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 26869760<>26873856 00000100:00100000:3.0:1789760776.130836:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.174914:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973879 gen 3 00000008:00010000:3.0:1789760776.174968:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760776.174976:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.174990:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.175014:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760776.175045:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.175182:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6560, 6560] 00000080:00200000:2.0:1789760776.175200:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.175207:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.175224:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26873856 i_blocks 53255 00000020:00200000:2.0:1789760776.175228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.175240:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.175275:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26873856, count: 4096 00000080:00200000:2.0:1789760776.175282:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26873856, bytes: 4096 00000080:00200000:2.0:1789760776.175292:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.175301:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.175304:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26873856, 26877952) 0 0 00000080:00200000:2.0:1789760776.175309:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.175326:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6561, 6561] 00020000:00200000:2.0:1789760776.175336:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26873856 chunk: [26873856, 26877952) 26877952, 4096 00020000:00200000:2.0:1789760776.175341:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.175362:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6561, 6561] 00000008:00010000:2.0:1789760776.175381:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.175388:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.175432:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26873856, 26877952) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.175447:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26873856=0x19a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.175502:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.175505:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26877952 00000020:00010000:2.0:1789760776.175513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.175516:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.175519:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.175523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.175526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.175533:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.175540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.175554:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760776.175656:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 26873856<>26877952 00000100:00100000:3.0:1789760776.175932:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.175955:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.243354:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973880 gen 3 00000008:00010000:3.0:1789760776.243438:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760776.243444:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.243453:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.243468:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760776.243492:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824768:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.243514:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6561, 6561] 00000080:00200000:2.0:1789760776.243537:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.243545:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.243563:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26877952 i_blocks 53255 00000020:00200000:2.0:1789760776.243568:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.243581:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.243614:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26877952, count: 4096 00000080:00200000:2.0:1789760776.243624:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26877952, bytes: 4096 00000080:00200000:2.0:1789760776.243633:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.243643:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.243645:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26877952, 26882048) 0 0 00000080:00200000:2.0:1789760776.243650:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.243668:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6562, 6562] 00020000:00200000:2.0:1789760776.243678:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26877952 chunk: [26877952, 26882048) 26882048, 4096 00020000:00200000:2.0:1789760776.243683:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.243704:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6562, 6562] 00000008:00010000:2.0:1789760776.243722:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.243729:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.243741:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26877952, 26882048) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.243756:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26877952=0x19a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.245303:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.245314:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26882048 00000020:00010000:2.0:1789760776.245327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.245331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.245336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.245340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.245343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.245354:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.245362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.245386:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760776.245519:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 26877952<>26882048 00000100:00100000:2.0:1789760776.245580:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824896:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.279627:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973881 gen 3 00000008:00010000:3.0:1789760776.279679:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760776.279688:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.279701:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.279722:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760776.279750:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493824896:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760776.279802:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6562, 6562] 00000080:00200000:1.0:1789760776.279824:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.279830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.279843:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26882048 i_blocks 53255 00000020:00200000:1.0:1789760776.279848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.279861:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.279889:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26882048, count: 4096 00000080:00200000:1.0:1789760776.279896:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26882048, bytes: 4096 00000080:00200000:1.0:1789760776.279907:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.279914:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.279916:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26882048, 26886144) 0 0 00000080:00200000:1.0:1789760776.279920:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.279933:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6563, 6563] 00020000:00200000:1.0:1789760776.279940:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26882048 chunk: [26882048, 26886144) 26886144, 4096 00020000:00200000:1.0:1789760776.279943:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.279960:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6563, 6563] 00000008:00010000:1.0:1789760776.279973:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.279978:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760776.279987:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26882048, 26886144) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.279998:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26882048=0x19a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.280052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.280054:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26886144 00000020:00010000:1.0:1789760776.280061:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.280065:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.280068:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.280080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.280083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.280092:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.280098:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.280122:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760776.280219:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 26882048<>26886144 00000100:00100000:1.0:1789760776.280283:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760776.280303:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493825024:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760776.318341:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973882 gen 3 00000008:00010000:1.0:1789760776.318390:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760776.318423:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.318433:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.318459:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760776.318485:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493825024:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.319055:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6563, 6563] 00000080:00200000:2.0:1789760776.319077:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.319083:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.319094:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26886144 i_blocks 53255 00000020:00200000:2.0:1789760776.319098:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.319107:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.319132:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26886144, count: 4096 00000080:00200000:2.0:1789760776.319138:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26886144, bytes: 4096 00000080:00200000:2.0:1789760776.319150:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.319157:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.319159:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26886144, 26890240) 0 0 00000080:00200000:2.0:1789760776.319163:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.319176:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6564, 6564] 00020000:00200000:2.0:1789760776.319183:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26886144 chunk: [26886144, 26890240) 26890240, 4096 00020000:00200000:2.0:1789760776.319187:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.319202:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6564, 6564] 00000008:00010000:2.0:1789760776.319215:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.319219:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.319229:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26886144, 26890240) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.319240:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26886144=0x19a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.319287:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.319290:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26890240 00000020:00010000:2.0:1789760776.319296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.319299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.319302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.319304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.319306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.319312:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.319318:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.319330:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760776.319461:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 26886144<>26890240 00000100:00100000:3.0:1789760776.319741:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.362196:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973883 gen 3 00000008:00010000:3.0:1789760776.362234:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760776.362241:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.362251:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.362277:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760776.362307:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.362445:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6564, 6564] 00000080:00200000:2.0:1789760776.362457:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.362462:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.362473:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26890240 i_blocks 53255 00000020:00200000:2.0:1789760776.362477:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.362487:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.362512:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26890240, count: 4096 00000080:00200000:2.0:1789760776.362518:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26890240, bytes: 4096 00000080:00200000:2.0:1789760776.362530:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.362538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.362540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26890240, 26894336) 0 0 00000080:00200000:2.0:1789760776.362544:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.362557:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6565, 6565] 00020000:00200000:2.0:1789760776.362564:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26890240 chunk: [26890240, 26894336) 26894336, 4096 00020000:00200000:2.0:1789760776.362567:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.362582:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6565, 6565] 00000008:00010000:2.0:1789760776.362595:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.362600:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.362609:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26890240, 26894336) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.362619:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26890240=0x19a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.362662:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.362665:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26894336 00000020:00010000:2.0:1789760776.362671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.362675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.362677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.362680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.362683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.362689:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.362695:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.362709:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760776.362785:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 26890240<>26894336 00000100:00100000:3.0:1789760776.362877:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493825280:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.401373:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973884 gen 3 00000008:00010000:3.0:1789760776.401509:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760776.401516:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.401526:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.401547:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760776.401574:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493825280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.401676:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6565, 6565] 00000080:00200000:2.0:1789760776.401688:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.401693:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.401704:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26894336 i_blocks 53255 00000020:00200000:2.0:1789760776.401708:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.401718:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.401745:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26894336, count: 4096 00000080:00200000:2.0:1789760776.401751:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26894336, bytes: 4096 00000080:00200000:2.0:1789760776.401760:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.401767:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.401770:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26894336, 26898432) 0 0 00000080:00200000:2.0:1789760776.401774:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.401788:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6566, 6566] 00020000:00200000:2.0:1789760776.401797:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26894336 chunk: [26894336, 26898432) 26898432, 4096 00020000:00200000:2.0:1789760776.401801:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.401815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6566, 6566] 00000008:00010000:2.0:1789760776.401828:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.401832:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.401842:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26894336, 26898432) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.401853:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26894336=0x19a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.401893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.401896:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26898432 00000020:00010000:2.0:1789760776.401902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.401906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.401909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.401912:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.401915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.401921:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.401927:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.401941:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760776.402021:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 26894336<>26898432 00000100:00100000:3.0:1789760776.403433:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.403459:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493825408:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760776.428620:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760776.464699:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973885 gen 3 00000008:00010000:3.0:1789760776.464753:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760776.464763:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.464776:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.464795:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760776.464822:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493825408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.465101:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6566, 6566] 00000080:00200000:2.0:1789760776.465126:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.465133:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.465149:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26898432 i_blocks 53255 00000020:00200000:2.0:1789760776.465152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.465163:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.465196:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26898432, count: 4096 00000080:00200000:2.0:1789760776.465204:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26898432, bytes: 4096 00000080:00200000:2.0:1789760776.465214:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.465223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.465226:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26898432, 26902528) 0 0 00000080:00200000:2.0:1789760776.465230:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.465256:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6567, 6567] 00020000:00200000:2.0:1789760776.465265:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26898432 chunk: [26898432, 26902528) 26902528, 4096 00020000:00200000:2.0:1789760776.465269:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.465291:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6567, 6567] 00000008:00010000:2.0:1789760776.465308:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.465315:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.465327:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26898432, 26902528) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.465342:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26898432=0x19a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.465481:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.465485:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26902528 00000020:00010000:2.0:1789760776.465494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.465498:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.465501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.465504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.465507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.465515:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.465522:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.465539:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760776.465730:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26898432<>26902528 00000100:00100000:3.0:1789760776.465779:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.465798:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825536:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760776.501066:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760776.501092:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.501104:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.501297:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760776.501330:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825536:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.501826:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6567, 6567] 00000080:00200000:2.0:1789760776.501842:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.501849:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.501864:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26902528 i_blocks 53255 00000020:00200000:2.0:1789760776.501868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.501879:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.501912:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26902528, count: 4096 00000080:00200000:2.0:1789760776.501919:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26902528, bytes: 4096 00000080:00200000:2.0:1789760776.501930:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.501940:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.501942:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26902528, 26906624) 0 0 00000080:00200000:2.0:1789760776.501947:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.501962:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6568, 6568] 00020000:00200000:2.0:1789760776.501970:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26902528 chunk: [26902528, 26906624) 26906624, 4096 00020000:00200000:2.0:1789760776.501975:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.501993:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6568, 6568] 00000008:00010000:2.0:1789760776.502009:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.502015:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.502026:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26902528, 26906624) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.502039:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26902528=0x19a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.502103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.502106:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26906624 00000020:00010000:2.0:1789760776.502114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.502118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.502120:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.502124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.502127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.502134:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.502141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.502156:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760776.502261:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26902528<>26906624 00000100:00100000:3.0:1789760776.504001:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825664:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.550503:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973887 gen 3 00000008:00010000:2.0:1789760776.553814:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:2.0:1789760776.553828:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.553840:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.553873:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:2.0:1789760776.553907:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493825664:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760776.554114:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6568, 6568] 00000080:00200000:1.0:1789760776.554131:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.554137:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.554152:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26906624 i_blocks 53255 00000020:00200000:1.0:1789760776.554156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.554166:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.554212:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26906624, count: 4096 00000080:00200000:1.0:1789760776.554220:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26906624, bytes: 4096 00000080:00200000:1.0:1789760776.554233:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.554523:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.554526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26906624, 26910720) 0 0 00000080:00200000:1.0:1789760776.554531:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.554548:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6569, 6569] 00020000:00200000:1.0:1789760776.554557:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26906624 chunk: [26906624, 26910720) 26910720, 4096 00020000:00200000:1.0:1789760776.554562:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.554583:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6569, 6569] 00000008:00010000:1.0:1789760776.554601:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.554607:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760776.554619:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26906624, 26910720) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.554636:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26906624=0x19a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.554692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.554695:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26910720 00000020:00010000:1.0:1789760776.554704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.554707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.554710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.554714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.554717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.554740:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.554746:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.554762:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760776.554866:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 26906624<>26910720 00000100:00100000:0.0:1789760776.555040:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493825792:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760776.581325:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973888 gen 3 00000008:00010000:0.0:1789760776.581364:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760776.581371:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.581379:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.581425:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760776.581449:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493825792:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760776.581504:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6569, 6569] 00000080:00200000:1.0:1789760776.581528:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.581535:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.581552:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26910720 i_blocks 53255 00000020:00200000:1.0:1789760776.581557:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.581571:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.581604:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26910720, count: 4096 00000080:00200000:1.0:1789760776.581611:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26910720, bytes: 4096 00000080:00200000:1.0:1789760776.581623:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.581633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.581636:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26910720, 26914816) 0 0 00000080:00200000:1.0:1789760776.581641:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.581658:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6570, 6570] 00020000:00200000:1.0:1789760776.581666:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26910720 chunk: [26910720, 26914816) 26914816, 4096 00020000:00200000:1.0:1789760776.581671:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.581691:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6570, 6570] 00000008:00010000:1.0:1789760776.581709:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.581715:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760776.581789:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26910720, 26914816) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.581806:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26910720=0x19aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.581870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.581873:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26914816 00000020:00010000:1.0:1789760776.581881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.581885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.581888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.581892:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.581895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.581905:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.581912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.581931:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760776.582031:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 26910720<>26914816 00000100:00100000:0.0:1789760776.582130:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493825920:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760776.616270:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973889 gen 3 00000008:00010000:0.0:1789760776.616307:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760776.616315:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.616323:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.616439:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760776.616465:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493825920:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760776.616561:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6570, 6570] 00000080:00200000:1.0:1789760776.616574:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.616580:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.616592:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26914816 i_blocks 53255 00000020:00200000:1.0:1789760776.616596:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.616605:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.616644:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26914816, count: 4096 00000080:00200000:1.0:1789760776.616651:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26914816, bytes: 4096 00000080:00200000:1.0:1789760776.616659:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.616666:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.616668:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26914816, 26918912) 0 0 00000080:00200000:1.0:1789760776.616672:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.616684:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6571, 6571] 00020000:00200000:1.0:1789760776.616691:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26914816 chunk: [26914816, 26918912) 26918912, 4096 00020000:00200000:1.0:1789760776.616695:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.616720:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6571, 6571] 00000008:00010000:1.0:1789760776.616735:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.616739:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760776.616748:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26914816, 26918912) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.616760:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26914816=0x19ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.616806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.616808:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26918912 00000020:00010000:1.0:1789760776.616815:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.616818:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.616821:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.616824:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.616826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.616835:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.616841:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.616856:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760776.616935:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 26914816<>26918912 00000100:00100000:0.0:1789760776.616994:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493826048:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760776.665357:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760776.665373:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.665386:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.665497:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760776.665527:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493826048:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760776.665594:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6571, 6571] 00000080:00200000:1.0:1789760776.665610:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.665616:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.665630:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26918912 i_blocks 53255 00000020:00200000:1.0:1789760776.665634:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.665644:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.665679:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26918912, count: 4096 00000080:00200000:1.0:1789760776.665686:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26918912, bytes: 4096 00000080:00200000:1.0:1789760776.665695:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.665704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.665706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26918912, 26923008) 0 0 00000080:00200000:1.0:1789760776.665711:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.665727:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6572, 6572] 00020000:00200000:1.0:1789760776.665735:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26918912 chunk: [26918912, 26923008) 26923008, 4096 00020000:00200000:1.0:1789760776.665740:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.665759:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6572, 6572] 00000008:00010000:1.0:1789760776.665776:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.665783:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760776.665794:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26918912, 26923008) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.665809:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26918912=0x19ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.665865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.665868:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26923008 00000020:00010000:1.0:1789760776.665877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.665880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.665883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.665887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.665890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.665900:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.665913:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.665929:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760776.666030:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 26918912<>26923008 00000100:00100000:0.0:1789760776.666102:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493826176:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760776.703058:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973891 gen 3 00000008:00010000:0.0:1789760776.703236:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760776.703244:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.703255:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.703330:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760776.703370:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493826176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.703742:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6572, 6572] 00000080:00200000:2.0:1789760776.703852:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.703957:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.703974:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26923008 i_blocks 53255 00000020:00200000:2.0:1789760776.703979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.703989:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.704022:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26923008, count: 4096 00000080:00200000:2.0:1789760776.704030:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26923008, bytes: 4096 00000080:00200000:2.0:1789760776.704042:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.704052:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.704054:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26923008, 26927104) 0 0 00000080:00200000:2.0:1789760776.704059:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.704087:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6573, 6573] 00020000:00200000:2.0:1789760776.704097:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26923008 chunk: [26923008, 26927104) 26927104, 4096 00020000:00200000:2.0:1789760776.704102:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.704258:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6573, 6573] 00000008:00010000:2.0:1789760776.704280:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.704286:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.704432:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26923008, 26927104) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.704448:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26923008=0x19ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.704510:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.704513:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26927104 00000020:00010000:2.0:1789760776.704522:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.704526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.704530:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.704533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.704536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.704545:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.704564:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.705043:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760776.705245:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26923008<>26927104 00000100:00100000:3.0:1789760776.705376:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493826304:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760776.750879:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760776.750896:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.750909:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.750931:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760776.750961:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493826304:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.751225:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6573, 6573] 00000080:00200000:2.0:1789760776.751243:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.751249:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.751265:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26927104 i_blocks 53255 00000020:00200000:2.0:1789760776.751268:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.751278:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.751313:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26927104, count: 4096 00000080:00200000:2.0:1789760776.751320:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26927104, bytes: 4096 00000080:00200000:2.0:1789760776.751330:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.751339:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.751341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26927104, 26931200) 0 0 00000080:00200000:2.0:1789760776.751346:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.751362:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6574, 6574] 00020000:00200000:2.0:1789760776.751370:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26927104 chunk: [26927104, 26931200) 26931200, 4096 00020000:00200000:2.0:1789760776.751375:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.751418:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6574, 6574] 00000008:00010000:2.0:1789760776.751437:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.751442:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.751455:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26927104, 26931200) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.751469:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26927104=0x19ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.751526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.751529:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26931200 00000020:00010000:2.0:1789760776.751536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.751540:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.751543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.751546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.751549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.751557:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.751564:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.751579:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760776.751681:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26927104<>26931200 00000100:00100000:3.0:1789760776.751827:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.788111:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973892 gen 3 00000100:00100000:3.0:1789760776.788208:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80700 x1876699493826432/t4294973893(4294973893) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760792 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760776.788259:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760776.788266:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.788277:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.788341:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760776.788366:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.788483:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6574, 6574] 00000080:00200000:2.0:1789760776.788499:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.788506:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.788522:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26931200 i_blocks 53255 00000020:00200000:2.0:1789760776.788526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.788537:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.788569:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26931200, count: 4096 00000080:00200000:2.0:1789760776.788577:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26931200, bytes: 4096 00000080:00200000:2.0:1789760776.788588:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.788597:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.788599:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26931200, 26935296) 0 0 00000080:00200000:2.0:1789760776.788603:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.788620:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6575, 6575] 00020000:00200000:2.0:1789760776.788629:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26931200 chunk: [26931200, 26935296) 26935296, 4096 00020000:00200000:2.0:1789760776.788634:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.788653:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6575, 6575] 00000008:00010000:2.0:1789760776.788670:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.788676:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.788688:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26931200, 26935296) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.788703:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26931200=0x19af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.788762:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.788765:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26935296 00000020:00010000:2.0:1789760776.788773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.788777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.788781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.788784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.788787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.788794:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.788802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.788871:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760776.788980:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26931200<>26935296 00000100:00100000:3.0:1789760776.789044:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493826560:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.826288:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973894 gen 3 00000008:00010000:3.0:1789760776.826359:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760776.826368:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.826380:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.826479:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760776.826503:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493826560:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.826614:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6575, 6575] 00000080:00200000:2.0:1789760776.826629:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.826633:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.826647:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26935296 i_blocks 53255 00000020:00200000:2.0:1789760776.826649:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.826659:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.826688:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26935296, count: 4096 00000080:00200000:2.0:1789760776.826694:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26935296, bytes: 4096 00000080:00200000:2.0:1789760776.826756:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.826765:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.826768:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26935296, 26939392) 0 0 00000080:00200000:2.0:1789760776.826773:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.826789:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6576, 6576] 00020000:00200000:2.0:1789760776.826797:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26935296 chunk: [26935296, 26939392) 26939392, 4096 00020000:00200000:2.0:1789760776.826801:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.826820:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6576, 6576] 00000008:00010000:2.0:1789760776.826837:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.826843:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.826855:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26935296, 26939392) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.826869:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26935296=0x19b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.826929:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.826932:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26939392 00000020:00010000:2.0:1789760776.826941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.826945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.826949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.826953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.826956:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.826964:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.826971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.826986:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760776.827170:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26935296<>26939392 00000100:00100000:3.0:1789760776.827237:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826688:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.869420:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973895 gen 3 00000008:00010000:3.0:1789760776.869486:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760776.869497:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.869509:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.869537:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760776.869574:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826688:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760776.869756:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6576, 6576] 00000080:00200000:0.0:1789760776.869773:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.869779:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.869797:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26939392 i_blocks 53255 00000020:00200000:0.0:1789760776.869801:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.869813:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.869857:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26939392, count: 4096 00000080:00200000:0.0:1789760776.869866:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26939392, bytes: 4096 00000080:00200000:0.0:1789760776.869875:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.869885:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.869888:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26939392, 26943488) 0 0 00000080:00200000:0.0:1789760776.869892:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.869908:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6577, 6577] 00020000:00200000:0.0:1789760776.869917:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26939392 chunk: [26939392, 26943488) 26943488, 4096 00020000:00200000:0.0:1789760776.869922:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.869941:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6577, 6577] 00000008:00010000:0.0:1789760776.869960:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.869968:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760776.869986:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26939392, 26943488) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.870001:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26939392=0x19b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.870060:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.870063:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26943488 00000020:00010000:0.0:1789760776.870081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.870085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.870088:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.870092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.870094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.870103:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.870111:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.870232:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760776.870335:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 26939392<>26943488 00000100:00100000:1.0:1789760776.870430:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493826816:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760776.901301:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760776.901317:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.901330:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.901359:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760776.901438:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493826816:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.901652:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6577, 6577] 00000080:00200000:2.0:1789760776.901669:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.901675:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.901691:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26943488 i_blocks 53255 00000020:00200000:2.0:1789760776.901694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.901705:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.901740:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26943488, count: 4096 00000080:00200000:2.0:1789760776.901748:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26943488, bytes: 4096 00000080:00200000:2.0:1789760776.901761:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.901770:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.901772:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26943488, 26947584) 0 0 00000080:00200000:2.0:1789760776.901776:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.901791:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6578, 6578] 00020000:00200000:2.0:1789760776.901799:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26943488 chunk: [26943488, 26947584) 26947584, 4096 00020000:00200000:2.0:1789760776.901803:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.901835:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6578, 6578] 00000008:00010000:2.0:1789760776.901854:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.901861:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.901873:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26943488, 26947584) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.901892:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26943488=0x19b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.901954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.901957:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26947584 00000020:00010000:2.0:1789760776.901966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.901969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.901973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.901976:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.901979:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.901988:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.901995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.902010:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760776.902278:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26943488<>26947584 00000100:00100000:3.0:1789760776.902620:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760776.902654:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.944841:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973897 gen 3 00000008:00010000:3.0:1789760776.944914:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760776.944923:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.944934:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.944954:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000080:00200000:2.0:1789760776.944980:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6578, 6578] 00000100:00100000:3.0:1789760776.944983:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493826944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.944997:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.945003:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.945019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26947584 i_blocks 53255 00000020:00200000:2.0:1789760776.945023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.945034:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.945082:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26947584, count: 4096 00000080:00200000:2.0:1789760776.945089:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26947584, bytes: 4096 00000080:00200000:2.0:1789760776.945099:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.945108:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.945110:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26947584, 26951680) 0 0 00000080:00200000:2.0:1789760776.945115:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.945132:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6579, 6579] 00020000:00200000:2.0:1789760776.945140:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26947584 chunk: [26947584, 26951680) 26951680, 4096 00020000:00200000:2.0:1789760776.945145:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.945166:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6579, 6579] 00000008:00010000:2.0:1789760776.945185:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.945192:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.945205:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26947584, 26951680) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.945221:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26947584=0x19b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.945284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.945287:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26951680 00000020:00010000:2.0:1789760776.945296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.945300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.945304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.945308:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.945310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.945319:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.945327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.945343:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760776.945493:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 26947584<>26951680 00000100:00100000:3.0:1789760776.946207:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.946233:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760776.979235:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973898 gen 3 00000008:00010000:3.0:1789760776.979284:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760776.979292:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.979304:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.979559:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760776.979588:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760776.979677:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6579, 6579] 00000080:00200000:2.0:1789760776.979693:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.979699:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.979713:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26951680 i_blocks 53255 00000020:00200000:2.0:1789760776.979717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.979727:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.979810:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26951680, count: 4096 00000080:00200000:2.0:1789760776.979817:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26951680, bytes: 4096 00000080:00200000:2.0:1789760776.979828:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.979837:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.979840:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26951680, 26955776) 0 0 00000080:00200000:2.0:1789760776.979845:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.979860:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6580, 6580] 00020000:00200000:2.0:1789760776.979868:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26951680 chunk: [26951680, 26955776) 26955776, 4096 00020000:00200000:2.0:1789760776.979873:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.979892:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6580, 6580] 00000008:00010000:2.0:1789760776.979911:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.979918:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760776.979930:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26951680, 26955776) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.979945:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26951680=0x19b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.980006:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.980010:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26955776 00000020:00010000:2.0:1789760776.980018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.980021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.980025:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.980028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.980031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.980040:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.980048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.980263:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760776.980632:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 26951680<>26955776 00000100:00100000:3.0:1789760776.980703:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.011304:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973899 gen 3 00000008:00010000:3.0:1789760777.011356:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760777.011364:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.011376:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.011414:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000080:00200000:2.0:1789760777.011525:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6580, 6580] 00000080:00200000:2.0:1789760777.011611:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.011618:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.011634:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26955776 i_blocks 53255 00000020:00200000:2.0:1789760777.011638:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.011650:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.011683:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26955776, count: 4096 00000080:00200000:2.0:1789760777.011692:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26955776, bytes: 4096 00000080:00200000:2.0:1789760777.011700:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.011709:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.011712:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26955776, 26959872) 0 0 00000080:00200000:2.0:1789760777.011716:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.011731:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6581, 6581] 00020000:00200000:2.0:1789760777.011740:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26955776 chunk: [26955776, 26959872) 26959872, 4096 00020000:00200000:2.0:1789760777.011745:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.011763:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6581, 6581] 00000008:00010000:2.0:1789760777.011782:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.011788:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760777.011800:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26955776, 26959872) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.011816:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26955776=0x19b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.011872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.011876:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26959872 00000020:00010000:2.0:1789760777.011883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.011887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.011891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.011894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.011897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.011904:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.011911:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760777.011956:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827200:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760777.012001:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:3.0:1789760777.012108:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 26955776<>26959872 00000100:00100000:3.0:1789760777.012308:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.012330:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827328:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.067294:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973900 gen 3 00000008:00010000:3.0:1789760777.067352:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760777.067361:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.067374:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.067485:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000080:00200000:2.0:1789760777.067492:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6581, 6581] 00000080:00200000:2.0:1789760777.067511:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760777.067514:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.067517:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.067533:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26959872 i_blocks 53255 00000020:00200000:2.0:1789760777.067537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.067549:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.067585:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26959872, count: 4096 00000080:00200000:2.0:1789760777.067592:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26959872, bytes: 4096 00000080:00200000:2.0:1789760777.067602:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.067611:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.067614:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26959872, 26963968) 0 0 00000080:00200000:2.0:1789760777.067619:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.067746:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6582, 6582] 00020000:00200000:2.0:1789760777.067756:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26959872 chunk: [26959872, 26963968) 26963968, 4096 00020000:00200000:2.0:1789760777.067760:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.067783:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6582, 6582] 00000008:00010000:2.0:1789760777.067801:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.067807:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760777.067820:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26959872, 26963968) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.067835:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26959872=0x19b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.067890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.067893:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26963968 00000020:00010000:2.0:1789760777.067902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.067905:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.067909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.067912:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.067916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.067923:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.067930:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.067944:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760777.068183:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 26959872<>26963968 00000100:00100000:2.0:1789760777.068229:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760777.068246:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827456:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.101327:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973901 gen 3 00000008:00010000:3.0:1789760777.101384:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760777.101439:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.101454:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.101514:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760777.101538:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.101686:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6582, 6582] 00000080:00200000:2.0:1789760777.101705:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.101711:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.101728:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26963968 i_blocks 53255 00000020:00200000:2.0:1789760777.101732:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.101743:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.101779:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26963968, count: 4096 00000080:00200000:2.0:1789760777.101787:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26963968, bytes: 4096 00000080:00200000:2.0:1789760777.101796:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.101806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.101808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26963968, 26968064) 0 0 00000080:00200000:2.0:1789760777.101813:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.101948:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6583, 6583] 00020000:00200000:2.0:1789760777.101956:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26963968 chunk: [26963968, 26968064) 26968064, 4096 00020000:00200000:2.0:1789760777.101960:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.101981:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6583, 6583] 00000008:00010000:2.0:1789760777.102000:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.102006:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760777.102019:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26963968, 26968064) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.102032:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26963968=0x19b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.102101:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.102104:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26968064 00000020:00010000:2.0:1789760777.102113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.102179:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.102185:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.102189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.102192:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.102202:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.102209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.102228:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760777.102329:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 26963968<>26968064 00000100:00100000:3.0:1789760777.102391:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827584:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.131520:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973902 gen 3 00000008:00010000:3.0:1789760777.131572:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760777.131581:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.131595:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.131615:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760777.131644:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.131654:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6583, 6583] 00000080:00200000:2.0:1789760777.131671:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.131677:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.131691:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26968064 i_blocks 53255 00000020:00200000:2.0:1789760777.131706:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.131717:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.131752:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26968064, count: 4096 00000080:00200000:2.0:1789760777.131759:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26968064, bytes: 4096 00000080:00200000:2.0:1789760777.131768:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.131777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.131779:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26968064, 26972160) 0 0 00000080:00200000:2.0:1789760777.131783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.131801:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6584, 6584] 00020000:00200000:2.0:1789760777.131809:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26968064 chunk: [26968064, 26972160) 26972160, 4096 00020000:00200000:2.0:1789760777.131813:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.131842:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6584, 6584] 00000008:00010000:2.0:1789760777.131868:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.131874:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760777.131886:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26968064, 26972160) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.131900:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26968064=0x19b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.131962:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.131965:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26972160 00000020:00010000:2.0:1789760777.131974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.131977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.131981:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.131985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.131987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.131995:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.132003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.132019:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760777.132131:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 26968064<>26972160 00000100:00100000:3.0:1789760777.132272:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.132289:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.185152:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973903 gen 3 00000008:00010000:3.0:1789760777.185205:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760777.185214:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.185226:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.185246:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760777.185273:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493827712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.185435:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6584, 6584] 00000080:00200000:2.0:1789760777.185453:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.185460:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.185476:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26972160 i_blocks 53255 00000020:00200000:2.0:1789760777.185479:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.185491:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.185526:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26972160, count: 4096 00000080:00200000:2.0:1789760777.185534:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26972160, bytes: 4096 00000080:00200000:2.0:1789760777.185553:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.185562:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.185564:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26972160, 26976256) 0 0 00000080:00200000:2.0:1789760777.185568:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.185711:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6585, 6585] 00020000:00200000:2.0:1789760777.185720:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26972160 chunk: [26972160, 26976256) 26976256, 4096 00020000:00200000:2.0:1789760777.185725:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.185746:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6585, 6585] 00000008:00010000:2.0:1789760777.185765:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.185771:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:2.0:1789760777.185783:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26972160, 26976256) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.185795:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26972160=0x19b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.185984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.185988:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26976256 00000020:00010000:2.0:1789760777.185998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.186001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.186005:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.186008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.186011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.186021:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.186028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.186047:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760777.186171:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 26972160<>26976256 00000100:00100000:3.0:1789760777.186333:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.186355:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.230964:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973904 gen 3 00000008:00010000:3.0:1789760777.231017:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760777.231025:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.231036:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.231054:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760777.231093:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.232216:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6585, 6585] 00000080:00200000:2.0:1789760777.232236:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.232243:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.232261:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26976256 i_blocks 53255 00000020:00200000:2.0:1789760777.232266:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.232278:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.232312:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26976256, count: 4096 00000080:00200000:2.0:1789760777.232319:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26976256, bytes: 4096 00000080:00200000:2.0:1789760777.232330:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.232339:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.232341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26976256, 26980352) 0 0 00000080:00200000:2.0:1789760777.232345:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.232362:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6586, 6586] 00020000:00200000:2.0:1789760777.232371:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26976256 chunk: [26976256, 26980352) 26980352, 4096 00020000:00200000:2.0:1789760777.232375:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.232430:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6586, 6586] 00000008:00010000:2.0:1789760777.232447:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.232454:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f030, osc lock ffff8eb650f45b50, flags 0x0 00000080:00200000:3.0:1789760777.232510:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26976256, 26980352) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.232525:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26976256=0x19ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.232660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.232664:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26980352 00000020:00010000:3.0:1789760777.232673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.232677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.232680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f030 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.232684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.232687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f030 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.232697:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.232704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.232720:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760777.232987:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 26976256<>26980352 00000100:00100000:2.0:1789760777.233061:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827968:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.262617:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973905 gen 3 00000008:00010000:3.0:1789760777.262673:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760777.262682:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.262692:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.262718:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760777.262746:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493827968:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.262951:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6586, 6586] 00000080:00200000:0.0:1789760777.262967:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.262974:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.262988:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26980352 i_blocks 53255 00000020:00200000:0.0:1789760777.262993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.263004:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.263038:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26980352, count: 4096 00000080:00200000:0.0:1789760777.263046:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26980352, bytes: 4096 00000080:00200000:0.0:1789760777.263056:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.263075:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.263078:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26980352, 26984448) 0 0 00000080:00200000:0.0:1789760777.263083:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.263099:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6587, 6587] 00020000:00200000:0.0:1789760777.263108:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26980352 chunk: [26980352, 26984448) 26984448, 4096 00020000:00200000:0.0:1789760777.263113:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.263290:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6587, 6587] 00000008:00010000:0.0:1789760777.263309:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.263315:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.263327:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26980352, 26984448) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.263341:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26980352=0x19bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.263497:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.263501:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26984448 00000020:00010000:0.0:1789760777.263509:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.263513:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.263517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.263521:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.263524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.263532:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.263539:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.263617:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760777.263716:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 26980352<>26984448 00000100:00100000:1.0:1789760777.263782:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828096:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.307218:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973906 gen 3 00000008:00010000:1.0:1789760777.307270:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760777.307279:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.307289:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.307355:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760777.307386:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828096:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.307795:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6587, 6587] 00000080:00200000:0.0:1789760777.307812:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.307870:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.307889:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26984448 i_blocks 53255 00000020:00200000:0.0:1789760777.307894:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.307906:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.307943:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26984448, count: 4096 00000080:00200000:0.0:1789760777.307950:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26984448, bytes: 4096 00000080:00200000:0.0:1789760777.307962:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.307971:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.307973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26984448, 26988544) 0 0 00000080:00200000:0.0:1789760777.307978:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.307995:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6588, 6588] 00020000:00200000:0.0:1789760777.308002:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26984448 chunk: [26984448, 26988544) 26988544, 4096 00020000:00200000:0.0:1789760777.308007:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.308026:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6588, 6588] 00000008:00010000:0.0:1789760777.308044:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.308051:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.308076:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26984448, 26988544) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.308090:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26984448=0x19bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.308278:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.308281:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26988544 00000020:00010000:0.0:1789760777.308290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.308294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.308297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.308301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.308304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.308313:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.308319:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.308339:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760777.308468:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 26984448<>26988544 00000100:00100000:1.0:1789760777.308610:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493828224:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760777.332951:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760777.332968:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.332978:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.333050:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760777.333090:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493828224:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.333221:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6588, 6588] 00000080:00200000:0.0:1789760777.333238:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.333244:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.333261:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26988544 i_blocks 53255 00000020:00200000:0.0:1789760777.333266:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.333277:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.333313:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26988544, count: 4096 00000080:00200000:0.0:1789760777.333321:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26988544, bytes: 4096 00000080:00200000:0.0:1789760777.333332:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.333341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.333344:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26988544, 26992640) 0 0 00000080:00200000:0.0:1789760777.333348:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.333364:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6589, 6589] 00020000:00200000:0.0:1789760777.333372:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26988544 chunk: [26988544, 26992640) 26992640, 4096 00020000:00200000:0.0:1789760777.333377:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.333429:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6589, 6589] 00000008:00010000:0.0:1789760777.333448:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.333454:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.333465:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26988544, 26992640) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.333478:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26988544=0x19bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.333535:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.333539:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26992640 00000020:00010000:0.0:1789760777.333624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.333629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.333632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.333635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.333638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.333647:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.333654:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.333671:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760777.333778:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 26988544<>26992640 00000100:00100000:1.0:1789760777.333856:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828352:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.360807:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973907 gen 3 00000100:00100000:1.0:1789760777.360864:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ca80 x1876699493828352/t4294973908(4294973908) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760793 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760777.360915:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760777.360923:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.360933:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.360963:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760777.360988:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828352:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.361283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6589, 6589] 00000080:00200000:0.0:1789760777.361302:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.361308:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.361324:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26992640 i_blocks 53255 00000020:00200000:0.0:1789760777.361328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.361337:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.361371:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26992640, count: 4096 00000080:00200000:0.0:1789760777.361378:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26992640, bytes: 4096 00000080:00200000:0.0:1789760777.361387:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.361424:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.361427:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26992640, 26996736) 0 0 00000080:00200000:0.0:1789760777.361432:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.361447:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6590, 6590] 00020000:00200000:0.0:1789760777.361457:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26992640 chunk: [26992640, 26996736) 26996736, 4096 00020000:00200000:0.0:1789760777.361462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.361482:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6590, 6590] 00000008:00010000:0.0:1789760777.361501:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.361507:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.361519:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26992640, 26996736) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.361533:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26992640=0x19be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.361655:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.361658:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26996736 00000020:00010000:0.0:1789760777.361666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.361669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.361673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.361676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.361679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.361688:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.361696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.361712:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760777.361823:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 26992640<>26996736 00000100:00100000:1.0:1789760777.361989:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493828480:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.395884:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973909 gen 3 00000008:00010000:1.0:1789760777.395950:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760777.395959:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.395970:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.396038:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760777.396083:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493828480:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.396272:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6590, 6590] 00000080:00200000:0.0:1789760777.396289:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.396296:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.396312:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26996736 i_blocks 53255 00000020:00200000:0.0:1789760777.396317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.396329:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.396366:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26996736, count: 4096 00000080:00200000:0.0:1789760777.396374:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26996736, bytes: 4096 00000080:00200000:0.0:1789760777.396388:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.396422:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.396425:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26996736, 27000832) 0 0 00000080:00200000:0.0:1789760777.396430:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.396447:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6591, 6591] 00020000:00200000:0.0:1789760777.396455:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26996736 chunk: [26996736, 27000832) 27000832, 4096 00020000:00200000:0.0:1789760777.396459:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.396480:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6591, 6591] 00000008:00010000:0.0:1789760777.396670:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.396677:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.396689:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26996736, 27000832) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.396703:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=26996736=0x19bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.396760:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.396764:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27000832 00000020:00010000:0.0:1789760777.396771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.396776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.396779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.396783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.396786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.396793:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.396800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.396816:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760777.396915:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 26996736<>27000832 00000100:00100000:1.0:1789760777.396989:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828608:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.425970:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973910 gen 3 00000008:00010000:1.0:1789760777.426025:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760777.426036:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.426047:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.426185:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760777.426217:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828608:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760777.426438:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6591, 6591] 00000080:00200000:3.0:1789760777.426465:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.426472:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.426489:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27000832 i_blocks 53255 00000020:00200000:3.0:1789760777.426494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.426505:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.426768:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27000832, count: 4096 00000080:00200000:3.0:1789760777.426776:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27000832, bytes: 4096 00000080:00200000:3.0:1789760777.426788:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.426798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.426800:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27000832, 27004928) 0 0 00000080:00200000:3.0:1789760777.426805:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.426822:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6592, 6592] 00020000:00200000:3.0:1789760777.426830:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27000832 chunk: [27000832, 27004928) 27004928, 4096 00020000:00200000:3.0:1789760777.426835:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.426856:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6592, 6592] 00000008:00010000:3.0:1789760777.426875:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.426882:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760777.426894:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27000832, 27004928) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.426907:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27000832=0x19c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.426968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.426971:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27004928 00000020:00010000:3.0:1789760777.426980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.426984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.426987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.426990:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.426993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.427000:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.427007:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.427025:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760777.427322:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 27000832<>27004928 00000100:00100000:2.0:1789760777.427452:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493828736:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760777.452817:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760777.463285:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973911 gen 3 00000008:00010000:3.0:1789760777.463339:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760777.463349:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.463359:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.463448:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760777.463478:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493828736:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.464473:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6592, 6592] 00000080:00200000:0.0:1789760777.464492:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.464498:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.464517:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27004928 i_blocks 53255 00000020:00200000:0.0:1789760777.464521:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.464533:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.464568:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27004928, count: 4096 00000080:00200000:0.0:1789760777.464576:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27004928, bytes: 4096 00000080:00200000:0.0:1789760777.464592:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.464603:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.464605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27004928, 27009024) 0 0 00000080:00200000:0.0:1789760777.464610:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.464626:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6593, 6593] 00020000:00200000:0.0:1789760777.464634:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27004928 chunk: [27004928, 27009024) 27009024, 4096 00020000:00200000:0.0:1789760777.464639:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.464660:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6593, 6593] 00000008:00010000:0.0:1789760777.464680:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.464686:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.464698:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27004928, 27009024) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.464713:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27004928=0x19c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.464776:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.464780:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27009024 00000020:00010000:0.0:1789760777.464787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.464791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.464794:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.464797:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.464800:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.464808:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.464815:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.464838:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760777.468920:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27004928<>27009024 00000100:00100000:1.0:1789760777.469004:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760777.469282:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828864:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760777.505703:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760777.505720:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.505732:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.505761:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760777.505792:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493828864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760777.505899:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6593, 6593] 00000080:00200000:3.0:1789760777.505926:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.505933:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.505950:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27009024 i_blocks 53255 00000020:00200000:3.0:1789760777.505954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.505965:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.506000:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27009024, count: 4096 00000080:00200000:3.0:1789760777.506012:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27009024, bytes: 4096 00000080:00200000:3.0:1789760777.506026:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.506035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.506037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27009024, 27013120) 0 0 00000080:00200000:3.0:1789760777.506042:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.506070:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6594, 6594] 00020000:00200000:3.0:1789760777.506080:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27009024 chunk: [27009024, 27013120) 27013120, 4096 00020000:00200000:3.0:1789760777.506086:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.506107:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6594, 6594] 00000008:00010000:3.0:1789760777.506131:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.506138:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760777.506150:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27009024, 27013120) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.506163:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27009024=0x19c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.506220:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.506223:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27013120 00000020:00010000:3.0:1789760777.506232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.506236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.506239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.506242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.506245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.506259:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.506265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.506287:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760777.506439:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 27009024<>27013120 00000100:00100000:3.0:1789760777.506483:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.506499:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493828992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.548474:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973913 gen 3 00000008:00010000:3.0:1789760777.548533:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:3.0:1789760777.548542:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.548551:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.548604:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:3.0:1789760777.548642:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493828992:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.548909:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6594, 6594] 00000080:00200000:2.0:1789760777.548925:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.548931:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.548946:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27013120 i_blocks 53255 00000020:00200000:2.0:1789760777.548949:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.548958:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.548989:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27013120, count: 4096 00000080:00200000:2.0:1789760777.548996:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27013120, bytes: 4096 00000080:00200000:2.0:1789760777.549006:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.549014:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.549017:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27013120, 27017216) 0 0 00000080:00200000:2.0:1789760777.549020:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.549037:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6595, 6595] 00020000:00200000:2.0:1789760777.549046:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27013120 chunk: [27013120, 27017216) 27017216, 4096 00020000:00200000:2.0:1789760777.549050:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.549081:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6595, 6595] 00000008:00010000:2.0:1789760777.549100:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.549167:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.549181:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27013120, 27017216) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.549194:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27013120=0x19c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.549250:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.549253:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27017216 00000020:00010000:2.0:1789760777.549261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.549264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.549267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.549270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.549274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.549347:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.549354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.549372:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760777.549512:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27013120<>27017216 00000100:00100000:3.0:1789760777.549578:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493829120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.598352:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973914 gen 3 00000008:00010000:3.0:1789760777.598430:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760777.598440:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.598453:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.598473:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760777.598502:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493829120:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.598977:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6595, 6595] 00000080:00200000:2.0:1789760777.598995:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.599002:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.599021:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27017216 i_blocks 53255 00000020:00200000:2.0:1789760777.599025:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.599037:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.599081:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27017216, count: 4096 00000080:00200000:2.0:1789760777.599089:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27017216, bytes: 4096 00000080:00200000:2.0:1789760777.599099:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.599107:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.599109:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27017216, 27021312) 0 0 00000080:00200000:2.0:1789760777.599114:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.599130:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6596, 6596] 00020000:00200000:2.0:1789760777.599139:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27017216 chunk: [27017216, 27021312) 27021312, 4096 00020000:00200000:2.0:1789760777.599144:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.599167:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6596, 6596] 00000008:00010000:2.0:1789760777.599185:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.599191:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.599204:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27017216, 27021312) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.599217:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27017216=0x19c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.599275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.599524:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27021312 00000020:00010000:2.0:1789760777.599533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.599538:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.599970:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.599974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.599977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.599987:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.599995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.600019:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760777.600142:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27017216<>27021312 00000100:00100000:3.0:1789760777.600822:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493829248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.633482:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973915 gen 3 00000008:00010000:3.0:1789760777.633532:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760777.633540:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.633551:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.633629:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760777.633656:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493829248:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.633936:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6596, 6596] 00000080:00200000:0.0:1789760777.633956:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.633962:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.633981:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27021312 i_blocks 53255 00000020:00200000:0.0:1789760777.633985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.633998:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.634037:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27021312, count: 4096 00000080:00200000:0.0:1789760777.634045:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27021312, bytes: 4096 00000080:00200000:0.0:1789760777.634567:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.634578:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.634581:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27021312, 27025408) 0 0 00000080:00200000:0.0:1789760777.634586:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.634603:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6597, 6597] 00020000:00200000:0.0:1789760777.634612:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27021312 chunk: [27021312, 27025408) 27025408, 4096 00020000:00200000:0.0:1789760777.634616:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.634704:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6597, 6597] 00000008:00010000:0.0:1789760777.634721:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.634728:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.634740:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27021312, 27025408) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.634754:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27021312=0x19c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.634885:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.634888:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27025408 00000020:00010000:0.0:1789760777.634896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.634900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.634904:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.634908:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.634911:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.634919:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.634926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.634948:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760777.635223:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27021312<>27025408 00000100:00100000:0.0:1789760777.635277:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.635298:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829376:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760777.678083:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760777.678099:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.678111:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.678139:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760777.678182:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829376:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760777.678370:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6597, 6597] 00000080:00200000:1.0:1789760777.678415:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.678421:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.678436:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27025408 i_blocks 53255 00000020:00200000:1.0:1789760777.678441:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.678451:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.678486:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27025408, count: 4096 00000080:00200000:1.0:1789760777.678492:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27025408, bytes: 4096 00000080:00200000:1.0:1789760777.678506:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.678515:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.678517:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27025408, 27029504) 0 0 00000080:00200000:1.0:1789760777.678522:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.678537:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6598, 6598] 00020000:00200000:1.0:1789760777.678545:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27025408 chunk: [27025408, 27029504) 27029504, 4096 00020000:00200000:1.0:1789760777.678550:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.678570:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6598, 6598] 00000008:00010000:1.0:1789760777.678589:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.678595:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760777.678607:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27025408, 27029504) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.678633:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27025408=0x19c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.678696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.678699:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27029504 00000020:00010000:1.0:1789760777.678708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.678712:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.678716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.678719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.678722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.678734:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.678741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.678756:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760777.678884:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 27025408<>27029504 00000100:00100000:1.0:1789760777.678930:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760777.678947:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493829504:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.724040:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973917 gen 3 00000008:00010000:1.0:1789760777.724094:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760777.724100:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.724105:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.724167:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000080:00200000:0.0:1789760777.724195:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6598, 6598] 00000100:00100000:1.0:1789760777.724197:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493829504:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760777.724208:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.724212:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.724224:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27029504 i_blocks 53255 00000020:00200000:0.0:1789760777.724228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.724236:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.724264:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27029504, count: 4096 00000080:00200000:0.0:1789760777.724271:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27029504, bytes: 4096 00000080:00200000:0.0:1789760777.724278:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.724284:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.724286:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27029504, 27033600) 0 0 00000080:00200000:0.0:1789760777.724289:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.724299:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6599, 6599] 00020000:00200000:0.0:1789760777.724305:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27029504 chunk: [27029504, 27033600) 27033600, 4096 00020000:00200000:0.0:1789760777.724307:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.724320:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6599, 6599] 00000008:00010000:0.0:1789760777.724330:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.724334:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760777.724341:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27029504, 27033600) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.724349:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27029504=0x19c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.724471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.724474:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27033600 00000020:00010000:0.0:1789760777.724481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.724484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.724487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.724490:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.724494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.724500:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.724506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.724520:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760777.724600:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 27029504<>27033600 00000100:00100000:0.0:1789760777.725201:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.725219:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829632:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760777.773186:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493829760:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.773297:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493830016:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.773339:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760777.773347:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829888:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.777371:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829888:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.777389:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493829760:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.777451:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493830016:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760777.788787:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973918 gen 3 00000008:00010000:1.0:1789760777.788827:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760777.788833:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.788841:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.788867:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760777.788889:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493829632:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.789185:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6599, 6599] 00000080:00200000:2.0:1789760777.789198:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.789203:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.789215:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27033600 i_blocks 53255 00000020:00200000:2.0:1789760777.789218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.789228:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.789255:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27033600, count: 4096 00000080:00200000:2.0:1789760777.789261:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27033600, bytes: 4096 00000080:00200000:2.0:1789760777.789270:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.789277:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.789280:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27033600, 27037696) 0 0 00000080:00200000:2.0:1789760777.789283:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.789295:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6600, 6600] 00020000:00200000:2.0:1789760777.789302:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27033600 chunk: [27033600, 27037696) 27037696, 4096 00020000:00200000:2.0:1789760777.789305:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.789320:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6600, 6600] 00000008:00010000:2.0:1789760777.789332:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.789338:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.789347:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27033600, 27037696) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.789359:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27033600=0x19c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.789437:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.789440:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27037696 00000020:00010000:2.0:1789760777.789446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.789449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.789451:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.789454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.789456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.789462:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.789467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.789481:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760777.789808:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27033600<>27037696 00000100:00100000:3.0:1789760777.789886:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.836941:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973919 gen 3 00000008:00010000:3.0:1789760777.836984:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760777.836990:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.836999:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.837018:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760777.837045:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.837166:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6600, 6600] 00000080:00200000:2.0:1789760777.837179:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.837184:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.837197:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27037696 i_blocks 53255 00000020:00200000:2.0:1789760777.837200:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.837209:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.837236:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27037696, count: 4096 00000080:00200000:2.0:1789760777.837242:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27037696, bytes: 4096 00000080:00200000:2.0:1789760777.837250:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.837258:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.837260:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27037696, 27041792) 0 0 00000080:00200000:2.0:1789760777.837264:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.837276:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6601, 6601] 00020000:00200000:2.0:1789760777.837284:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27037696 chunk: [27037696, 27041792) 27041792, 4096 00020000:00200000:2.0:1789760777.837287:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.837302:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6601, 6601] 00000008:00010000:2.0:1789760777.837314:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.837319:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.837328:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27037696, 27041792) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.837339:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27037696=0x19c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.837381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.837384:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27041792 00000020:00010000:2.0:1789760777.837390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.837438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.837441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.837444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.837447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.837454:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.837460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.837473:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760777.837549:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27037696<>27041792 00000100:00100000:3.0:1789760777.837717:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.874520:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973920 gen 3 00000008:00010000:3.0:1789760777.874612:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760777.874619:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.874627:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.874730:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760777.874757:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830272:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.874849:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6601, 6601] 00000080:00200000:2.0:1789760777.874862:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.874867:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.874879:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27041792 i_blocks 53255 00000020:00200000:2.0:1789760777.874882:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.874891:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.874918:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27041792, count: 4096 00000080:00200000:2.0:1789760777.874924:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27041792, bytes: 4096 00000080:00200000:2.0:1789760777.874932:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.874938:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.874941:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27041792, 27045888) 0 0 00000080:00200000:2.0:1789760777.874944:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.874957:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6602, 6602] 00020000:00200000:2.0:1789760777.874964:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27041792 chunk: [27041792, 27045888) 27045888, 4096 00020000:00200000:2.0:1789760777.874967:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.874982:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6602, 6602] 00000008:00010000:2.0:1789760777.874995:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.875000:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.875009:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27041792, 27045888) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.875020:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27041792=0x19ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.875082:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.875085:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27045888 00000020:00010000:2.0:1789760777.875101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.875105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.875109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.875111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.875122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.875129:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.875135:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.875158:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b600 00000008:00100000:2.0:1789760777.875244:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27041792<>27045888 00000100:00100000:3.0:1789760777.875492:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760777.875507:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830400:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.914803:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973921 gen 3 00000008:00010000:3.0:1789760777.914840:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b600 00000020:00200000:3.0:1789760777.914847:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.914856:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.914876:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b600 00000100:00100000:3.0:1789760777.914910:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830400:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.915257:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6602, 6602] 00000080:00200000:2.0:1789760777.915271:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.915276:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.915288:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27045888 i_blocks 53255 00000020:00200000:2.0:1789760777.915291:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.915300:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.915329:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27045888, count: 4096 00000080:00200000:2.0:1789760777.915335:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27045888, bytes: 4096 00000080:00200000:2.0:1789760777.915344:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.915351:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.915353:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27045888, 27049984) 0 0 00000080:00200000:2.0:1789760777.915357:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.915370:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6603, 6603] 00020000:00200000:2.0:1789760777.915376:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27045888 chunk: [27045888, 27049984) 27049984, 4096 00020000:00200000:2.0:1789760777.915380:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.915425:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6603, 6603] 00000008:00010000:2.0:1789760777.915439:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.915443:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.915452:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27045888, 27049984) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.915463:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27045888=0x19cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.915505:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.915508:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27049984 00000020:00010000:2.0:1789760777.915514:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.915517:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.915520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.915523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.915526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.915532:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.915537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.915551:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760777.915626:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27045888<>27049984 00000100:00100000:3.0:1789760777.916433:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.916456:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830528:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760777.945312:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760777.945325:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.945334:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.945432:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760777.945458:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830528:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.945601:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6603, 6603] 00000080:00200000:2.0:1789760777.945612:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.945617:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.945628:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27049984 i_blocks 53255 00000020:00200000:2.0:1789760777.945632:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.945640:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.945666:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27049984, count: 4096 00000080:00200000:2.0:1789760777.945672:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27049984, bytes: 4096 00000080:00200000:2.0:1789760777.945680:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.945687:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.945689:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27049984, 27054080) 0 0 00000080:00200000:2.0:1789760777.945693:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.945706:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6604, 6604] 00020000:00200000:2.0:1789760777.945712:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27049984 chunk: [27049984, 27054080) 27054080, 4096 00020000:00200000:2.0:1789760777.945716:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.945731:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6604, 6604] 00000008:00010000:2.0:1789760777.945745:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.945749:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.945758:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27049984, 27054080) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.945769:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27049984=0x19cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.945810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.945813:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27054080 00000020:00010000:2.0:1789760777.945820:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.945823:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.945826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.945829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.945831:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.945838:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.945844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.945858:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760777.945935:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 27049984<>27054080 00000100:00100000:3.0:1789760777.945990:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760777.983768:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973922 gen 3 00000100:00100000:3.0:1789760777.983799:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80380 x1876699493830656/t4294973923(4294973923) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760793 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760777.983848:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760777.983854:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760777.983862:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760777.983881:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760777.983903:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493830656:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760777.983990:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6604, 6604] 00000080:00200000:2.0:1789760777.984003:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.984008:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.984019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27054080 i_blocks 53255 00000020:00200000:2.0:1789760777.984023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.984032:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.984069:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27054080, count: 4096 00000080:00200000:2.0:1789760777.984075:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27054080, bytes: 4096 00000080:00200000:2.0:1789760777.984083:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.984090:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.984092:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27054080, 27058176) 0 0 00000080:00200000:2.0:1789760777.984095:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.984107:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6605, 6605] 00020000:00200000:2.0:1789760777.984180:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27054080 chunk: [27054080, 27058176) 27058176, 4096 00020000:00200000:2.0:1789760777.984185:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.984200:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6605, 6605] 00000008:00010000:2.0:1789760777.984214:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.984218:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760777.984227:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27054080, 27058176) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.984237:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27054080=0x19cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.984279:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.984283:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27058176 00000020:00010000:2.0:1789760777.984289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.984292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.984295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.984298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.984300:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.984307:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.984313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.984328:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760777.984445:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 27054080<>27058176 00000100:00100000:3.0:1789760777.984536:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830784:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760778.022550:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973924 gen 3 00000008:00010000:3.0:1789760778.022594:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760778.022602:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.022610:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.022629:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760778.022653:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830784:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.022781:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6605, 6605] 00000080:00200000:2.0:1789760778.022794:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.022799:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.022813:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27058176 i_blocks 53255 00000020:00200000:2.0:1789760778.022827:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.022837:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.022864:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27058176, count: 4096 00000080:00200000:2.0:1789760778.022870:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27058176, bytes: 4096 00000080:00200000:2.0:1789760778.022878:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.022885:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.022887:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27058176, 27062272) 0 0 00000080:00200000:2.0:1789760778.022890:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.022903:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6606, 6606] 00020000:00200000:2.0:1789760778.022910:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27058176 chunk: [27058176, 27062272) 27062272, 4096 00020000:00200000:2.0:1789760778.022914:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.022930:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6606, 6606] 00000008:00010000:2.0:1789760778.022943:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.022949:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.022957:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27058176, 27062272) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.022968:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27058176=0x19ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.023013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.023016:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27062272 00000020:00010000:2.0:1789760778.023023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.023027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.023029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.023032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.023035:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.023040:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.023046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.023071:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760778.023146:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27058176<>27062272 00000100:00100000:3.0:1789760778.023319:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.023334:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830912:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760778.057871:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760778.057881:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.057891:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.057909:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760778.057932:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493830912:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.058018:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6606, 6606] 00000080:00200000:2.0:1789760778.058029:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.058034:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.058050:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27062272 i_blocks 53255 00000020:00200000:2.0:1789760778.058053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.058070:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.058096:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27062272, count: 4096 00000080:00200000:2.0:1789760778.058102:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27062272, bytes: 4096 00000080:00200000:2.0:1789760778.058109:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.058185:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.058187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27062272, 27066368) 0 0 00000080:00200000:2.0:1789760778.058191:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.058204:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6607, 6607] 00020000:00200000:2.0:1789760778.058211:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27062272 chunk: [27062272, 27066368) 27066368, 4096 00020000:00200000:2.0:1789760778.058215:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.058230:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6607, 6607] 00000008:00010000:2.0:1789760778.058243:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.058248:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.058256:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27062272, 27066368) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.058266:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27062272=0x19cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.058311:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.058314:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27066368 00000020:00010000:2.0:1789760778.058320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.058323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.058326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.058329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.058331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.058338:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.058343:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.058357:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760778.058593:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27062272<>27066368 00000100:00100000:2.0:1789760778.058635:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493831040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760778.103353:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973926 gen 3 00000008:00010000:3.0:1789760778.103425:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760778.103431:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760778.103440:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760778.103528:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760778.103561:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493831040:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.103800:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6607, 6607] 00000080:00200000:1.0:1789760778.103812:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.104089:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.104103:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27066368 i_blocks 53255 00000020:00200000:1.0:1789760778.104107:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.104188:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.104213:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27066368, count: 4096 00000080:00200000:1.0:1789760778.104219:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27066368, bytes: 4096 00000080:00200000:1.0:1789760778.104229:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.104237:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.104239:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27066368, 27070464) 0 0 00000080:00200000:1.0:1789760778.104242:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.104255:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6608, 6608] 00020000:00200000:1.0:1789760778.104261:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27066368 chunk: [27066368, 27070464) 27070464, 4096 00020000:00200000:1.0:1789760778.104265:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.104279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6608, 6608] 00000008:00010000:1.0:1789760778.104293:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.104297:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.104306:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27066368, 27070464) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.104318:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27066368=0x19d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.104369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.104371:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27070464 00000020:00010000:1.0:1789760778.104379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.104382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.104385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.104388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.104391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.104440:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.104444:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.104459:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760778.104600:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 27066368<>27070464 00000100:00100000:1.0:1789760778.104635:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.104649:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831168:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760778.149912:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760778.149923:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.149931:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.149955:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760778.149981:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831168:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760778.150159:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6608, 6608] 00000080:00200000:0.0:1789760778.150170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.150176:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.150187:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27070464 i_blocks 53255 00000020:00200000:0.0:1789760778.150190:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.150199:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.150225:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27070464, count: 4096 00000080:00200000:0.0:1789760778.150230:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27070464, bytes: 4096 00000080:00200000:0.0:1789760778.150238:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.150245:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.150247:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27070464, 27074560) 0 0 00000080:00200000:0.0:1789760778.150251:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.150263:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6609, 6609] 00020000:00200000:0.0:1789760778.150274:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27070464 chunk: [27070464, 27074560) 27074560, 4096 00020000:00200000:0.0:1789760778.150278:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.150293:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6609, 6609] 00000008:00010000:0.0:1789760778.150308:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.150312:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760778.150321:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27070464, 27074560) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.150332:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27070464=0x19d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.150388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.150391:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27074560 00000020:00010000:0.0:1789760778.150441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.150444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.150447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.150450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.150453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.150460:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.150465:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.150478:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760778.150838:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 27070464<>27074560 00000100:00100000:0.0:1789760778.150934:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.150948:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493831296:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760778.192671:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973928 gen 3 00000008:00010000:0.0:1789760778.192716:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760778.192722:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.192730:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.192783:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760778.192800:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493831296:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.192919:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6609, 6609] 00000080:00200000:1.0:1789760778.192932:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.193139:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.193152:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27074560 i_blocks 53255 00000020:00200000:1.0:1789760778.193156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.193165:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.193191:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27074560, count: 4096 00000080:00200000:1.0:1789760778.193198:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27074560, bytes: 4096 00000080:00200000:1.0:1789760778.193205:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.193212:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.193214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27074560, 27078656) 0 0 00000080:00200000:1.0:1789760778.193218:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.193230:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6610, 6610] 00020000:00200000:1.0:1789760778.193236:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27074560 chunk: [27074560, 27078656) 27078656, 4096 00020000:00200000:1.0:1789760778.193240:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.193255:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6610, 6610] 00000008:00010000:1.0:1789760778.193267:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.193271:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.193280:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27074560, 27078656) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.193290:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27074560=0x19d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.193329:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.193331:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27078656 00000020:00010000:1.0:1789760778.193337:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.193341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.193344:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.193347:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.193349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.193356:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.193361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.193375:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760778.193471:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 27074560<>27078656 00000100:00100000:0.0:1789760778.193530:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493831424:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760778.237609:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973929 gen 3 00000008:00010000:0.0:1789760778.237646:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760778.237653:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.237660:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.237681:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760778.237706:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493831424:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.237917:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6610, 6610] 00000080:00200000:1.0:1789760778.237931:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.237937:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.237949:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27078656 i_blocks 53255 00000020:00200000:1.0:1789760778.237953:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.237961:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.238068:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27078656, count: 4096 00000080:00200000:1.0:1789760778.238075:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27078656, bytes: 4096 00000080:00200000:1.0:1789760778.238082:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.238089:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.238091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27078656, 27082752) 0 0 00000080:00200000:1.0:1789760778.238095:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.238178:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6611, 6611] 00020000:00200000:1.0:1789760778.238187:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27078656 chunk: [27078656, 27082752) 27082752, 4096 00020000:00200000:1.0:1789760778.238191:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.238208:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6611, 6611] 00000008:00010000:1.0:1789760778.238222:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.238227:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.238236:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27078656, 27082752) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.238248:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27078656=0x19d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.238292:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.238295:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27082752 00000020:00010000:1.0:1789760778.238303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.238307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.238310:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.238314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.238317:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.238326:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.238332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.238348:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.238471:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 27078656<>27082752 00000100:00100000:0.0:1789760778.238647:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831552:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760778.286805:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760778.286817:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.286825:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.286847:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760778.286870:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831552:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760778.288106:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6611, 6611] 00000080:00200000:3.0:1789760778.288122:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.288128:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760778.288140:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27082752 i_blocks 53255 00000020:00200000:3.0:1789760778.288143:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760778.288153:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760778.288180:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27082752, count: 4096 00000080:00200000:3.0:1789760778.288185:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27082752, bytes: 4096 00000080:00200000:3.0:1789760778.288196:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760778.288203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760778.288205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27082752, 27086848) 0 0 00000080:00200000:3.0:1789760778.288210:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760778.288223:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6612, 6612] 00020000:00200000:3.0:1789760778.288230:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27082752 chunk: [27082752, 27086848) 27086848, 4096 00020000:00200000:3.0:1789760778.288234:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760778.288250:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6612, 6612] 00000008:00010000:3.0:1789760778.288264:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760778.288269:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760778.288280:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27082752, 27086848) append 0 append lockpos 0 00000080:00200000:3.0:1789760778.288291:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27082752=0x19d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760778.288339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760778.288342:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27086848 00000020:00010000:3.0:1789760778.288348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:3.0:1789760778.288352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.288354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760778.288357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760778.288360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760778.288368:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760778.288374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760778.288391:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760778.288491:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 27082752<>27086848 00000100:00100000:3.0:1789760778.288538:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493831680:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.329240:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973931 gen 3 00000008:00010000:2.0:1789760778.329295:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760778.329303:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.329310:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.329384:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760778.329433:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493831680:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760778.329551:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6612, 6612] 00000080:00200000:3.0:1789760778.329563:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760778.329568:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.330318:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27086848 i_blocks 53255 00000020:00200000:2.0:1789760778.330326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.330335:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.330366:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27086848, count: 4096 00000080:00200000:2.0:1789760778.330373:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27086848, bytes: 4096 00000080:00200000:2.0:1789760778.330382:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.330391:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.330421:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27086848, 27090944) 0 0 00000080:00200000:2.0:1789760778.330425:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.330440:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6613, 6613] 00020000:00200000:2.0:1789760778.330447:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27086848 chunk: [27086848, 27090944) 27090944, 4096 00020000:00200000:2.0:1789760778.330451:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.330468:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6613, 6613] 00000008:00010000:2.0:1789760778.330484:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.330489:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.330498:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27086848, 27090944) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.330509:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27086848=0x19d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.330556:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.330559:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27090944 00000020:00010000:2.0:1789760778.330565:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.330568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.330570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.330572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.330575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.330580:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.330586:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.330631:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760778.330708:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27086848<>27090944 00000100:00100000:3.0:1789760778.330772:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.330900:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493831808:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.369281:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973932 gen 3 00000008:00010000:2.0:1789760778.369316:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760778.369322:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.369330:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.369425:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760778.369447:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493831808:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.369568:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6613, 6613] 00000080:00200000:1.0:1789760778.369580:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.369586:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.369599:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27090944 i_blocks 53255 00000020:00200000:1.0:1789760778.369602:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.369672:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.369700:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27090944, count: 4096 00000080:00200000:1.0:1789760778.369705:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27090944, bytes: 4096 00000080:00200000:1.0:1789760778.369713:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.369720:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.369722:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27090944, 27095040) 0 0 00000080:00200000:1.0:1789760778.369726:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.369738:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6614, 6614] 00020000:00200000:1.0:1789760778.369745:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27090944 chunk: [27090944, 27095040) 27095040, 4096 00020000:00200000:1.0:1789760778.369748:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.369762:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6614, 6614] 00000008:00010000:1.0:1789760778.369775:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.369779:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.369787:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27090944, 27095040) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.369797:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27090944=0x19d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.369838:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.369840:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27095040 00000020:00010000:1.0:1789760778.369846:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.369849:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.369852:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.369855:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.369858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.369865:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.369870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.369883:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.370024:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27090944<>27095040 00000100:00100000:1.0:1789760778.370074:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.370090:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831936:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760778.418648:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973933 gen 3 00000008:00010000:1.0:1789760778.418687:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760778.418704:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.418715:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.418735:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760778.418760:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493831936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.418922:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6614, 6614] 00000080:00200000:2.0:1789760778.418935:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.418940:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.418952:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27095040 i_blocks 53255 00000020:00200000:2.0:1789760778.418956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.418965:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.418993:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27095040, count: 4096 00000080:00200000:2.0:1789760778.419000:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27095040, bytes: 4096 00000080:00200000:2.0:1789760778.419008:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.419016:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.419018:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27095040, 27099136) 0 0 00000080:00200000:2.0:1789760778.419022:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.419053:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6615, 6615] 00020000:00200000:2.0:1789760778.419070:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27095040 chunk: [27095040, 27099136) 27099136, 4096 00020000:00200000:2.0:1789760778.419074:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.419090:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6615, 6615] 00000008:00010000:2.0:1789760778.419108:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.419112:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.419121:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27095040, 27099136) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.419132:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27095040=0x19d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.419178:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.419180:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27099136 00000020:00010000:2.0:1789760778.419198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.419202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.419204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.419207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.419210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.419217:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.419222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.419238:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760778.419318:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27095040<>27099136 00000100:00100000:2.0:1789760778.419362:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832064:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.475823:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973934 gen 3 00000008:00010000:2.0:1789760778.475859:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760778.475866:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.475877:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.475902:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760778.475928:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832064:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.476770:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6615, 6615] 00000080:00200000:1.0:1789760778.476785:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.476791:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.476803:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27099136 i_blocks 53255 00000020:00200000:1.0:1789760778.476808:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.476819:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.476847:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27099136, count: 4096 00000080:00200000:1.0:1789760778.476854:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27099136, bytes: 4096 00000080:00200000:1.0:1789760778.476865:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.476872:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.476874:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27099136, 27103232) 0 0 00000080:00200000:1.0:1789760778.476878:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.476890:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6616, 6616] 00020000:00200000:1.0:1789760778.476898:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27099136 chunk: [27099136, 27103232) 27103232, 4096 00020000:00200000:1.0:1789760778.476902:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.476919:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6616, 6616] 00000008:00010000:1.0:1789760778.476933:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.476937:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.476947:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27099136, 27103232) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.476960:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27099136=0x19d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.479694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.479704:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27103232 00000020:00010000:1.0:1789760778.479716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.479721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.479724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.479727:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.479730:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.479741:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.479750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.479784:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.479905:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27099136<>27103232 00010000:00010000:0.0:1789760778.480083:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760778.480104:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832192:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760778.526257:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973935 gen 3 00000008:00010000:1.0:1789760778.526298:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760778.526307:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.526315:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.526341:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760778.526370:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.526812:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6616, 6616] 00000080:00200000:2.0:1789760778.526825:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.526830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.526842:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27103232 i_blocks 53255 00000020:00200000:2.0:1789760778.526845:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.526855:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.526884:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27103232, count: 4096 00000080:00200000:2.0:1789760778.526891:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27103232, bytes: 4096 00000080:00200000:2.0:1789760778.526900:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.526908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.526910:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27103232, 27107328) 0 0 00000080:00200000:2.0:1789760778.526914:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.526927:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6617, 6617] 00020000:00200000:2.0:1789760778.526935:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27103232 chunk: [27103232, 27107328) 27107328, 4096 00020000:00200000:2.0:1789760778.526939:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.526954:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6617, 6617] 00000008:00010000:2.0:1789760778.526966:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.526971:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.526980:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27103232, 27107328) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.526991:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27103232=0x19d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.527036:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.527039:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27107328 00000020:00010000:2.0:1789760778.527046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.527049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.527052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.527066:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.527069:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.527077:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.527082:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.527098:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760778.527175:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27103232<>27107328 00000100:00100000:2.0:1789760778.527214:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760778.527229:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832320:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.596319:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973936 gen 3 00000008:00010000:2.0:1789760778.596355:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:2.0:1789760778.596361:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.596369:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.596429:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:2.0:1789760778.596456:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832320:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.596535:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6617, 6617] 00000080:00200000:1.0:1789760778.596549:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.596554:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.596567:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27107328 i_blocks 53255 00000020:00200000:1.0:1789760778.596571:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.596580:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.596610:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27107328, count: 4096 00000080:00200000:1.0:1789760778.596616:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27107328, bytes: 4096 00000080:00200000:1.0:1789760778.596624:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.596632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.596635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27107328, 27111424) 0 0 00000080:00200000:1.0:1789760778.596639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.596651:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6618, 6618] 00020000:00200000:1.0:1789760778.596658:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27107328 chunk: [27107328, 27111424) 27111424, 4096 00020000:00200000:1.0:1789760778.596662:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.596676:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6618, 6618] 00000008:00010000:1.0:1789760778.596690:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.596695:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.596704:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27107328, 27111424) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.596715:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27107328=0x19da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.596765:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.596767:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27111424 00000020:00010000:1.0:1789760778.596774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.596777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.596780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.596783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.596786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.596794:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.596800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.596815:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.596897:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27107328<>27111424 00000100:00100000:1.0:1789760778.596939:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.596957:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832448:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760778.650900:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760778.650911:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.650920:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.650944:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760778.650970:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832448:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.651162:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6618, 6618] 00000080:00200000:2.0:1789760778.651176:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.651181:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.651192:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27111424 i_blocks 53255 00000020:00200000:2.0:1789760778.651195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.651204:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.651232:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27111424, count: 4096 00000080:00200000:2.0:1789760778.651239:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27111424, bytes: 4096 00000080:00200000:2.0:1789760778.651248:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.651255:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.651257:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27111424, 27115520) 0 0 00000080:00200000:2.0:1789760778.651261:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.651273:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6619, 6619] 00020000:00200000:2.0:1789760778.651280:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27111424 chunk: [27111424, 27115520) 27115520, 4096 00020000:00200000:2.0:1789760778.651284:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.651300:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6619, 6619] 00000008:00010000:2.0:1789760778.651314:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.651318:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.651328:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27111424, 27115520) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.651338:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27111424=0x19db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.651377:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.651379:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27115520 00000020:00010000:2.0:1789760778.651385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.651388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.651390:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.651419:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.651421:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.651427:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.651432:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.651921:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:1.0:1789760778.652089:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 27111424<>27115520 00000100:00100000:0.0:1789760778.652148:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.652298:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493832576:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760778.693338:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973938 gen 3 00000008:00010000:0.0:1789760778.693388:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:0.0:1789760778.693426:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.693445:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.693465:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:0.0:1789760778.693490:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493832576:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.693647:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6619, 6619] 00000080:00200000:2.0:1789760778.693659:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.693664:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.693674:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27115520 i_blocks 53255 00000020:00200000:2.0:1789760778.693677:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.693686:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.693710:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27115520, count: 4096 00000080:00200000:2.0:1789760778.693716:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27115520, bytes: 4096 00000080:00200000:2.0:1789760778.693723:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.693730:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.693732:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27115520, 27119616) 0 0 00000080:00200000:2.0:1789760778.693735:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.693748:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6620, 6620] 00020000:00200000:2.0:1789760778.693755:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27115520 chunk: [27115520, 27119616) 27119616, 4096 00020000:00200000:2.0:1789760778.693759:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.693772:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6620, 6620] 00000008:00010000:2.0:1789760778.693784:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.693789:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.693798:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27115520, 27119616) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.693807:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27115520=0x19dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.693849:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.693851:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27119616 00000020:00010000:2.0:1789760778.693858:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.693861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.693864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.693867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.693870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.693876:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.693881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.693907:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760778.693981:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27115520<>27119616 00000100:00100000:3.0:1789760778.694226:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760778.695838:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493832704:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.735210:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973939 gen 3 00000008:00010000:2.0:1789760778.735245:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760778.735252:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.735261:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.735286:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760778.735310:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493832704:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.735418:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6620, 6620] 00000080:00200000:1.0:1789760778.735443:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.735448:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.735460:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27119616 i_blocks 53255 00000020:00200000:1.0:1789760778.735465:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.735475:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.735503:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27119616, count: 4096 00000080:00200000:1.0:1789760778.735510:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27119616, bytes: 4096 00000080:00200000:1.0:1789760778.735519:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.735526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.735528:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27119616, 27123712) 0 0 00000080:00200000:1.0:1789760778.735533:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.735556:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6621, 6621] 00020000:00200000:1.0:1789760778.735564:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27119616 chunk: [27119616, 27123712) 27123712, 4096 00020000:00200000:1.0:1789760778.735568:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.735586:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6621, 6621] 00000008:00010000:1.0:1789760778.735600:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.735605:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.735615:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27119616, 27123712) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.735624:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27119616=0x19dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.735669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.735671:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27123712 00000020:00010000:1.0:1789760778.735678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.735681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.735684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.735687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.735689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.735698:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.735703:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.735719:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.735810:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27119616<>27123712 00000100:00100000:1.0:1789760778.735845:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.735858:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832832:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760778.776753:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973940 gen 3 00000008:00010000:1.0:1789760778.776793:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760778.776800:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.776808:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.776833:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760778.780512:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493832832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.780694:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6621, 6621] 00000080:00200000:2.0:1789760778.780707:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.780713:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.780726:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27123712 i_blocks 53255 00000020:00200000:2.0:1789760778.780730:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.780742:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.780770:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27123712, count: 4096 00000080:00200000:2.0:1789760778.780777:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27123712, bytes: 4096 00000080:00200000:2.0:1789760778.780788:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.780795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.780798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27123712, 27127808) 0 0 00000080:00200000:2.0:1789760778.780802:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.780814:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6622, 6622] 00020000:00200000:2.0:1789760778.780820:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27123712 chunk: [27123712, 27127808) 27127808, 4096 00020000:00200000:2.0:1789760778.780824:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.780840:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6622, 6622] 00000008:00010000:2.0:1789760778.780853:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.780857:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.780865:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27123712, 27127808) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.780875:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27123712=0x19de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.780922:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.780925:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27127808 00000020:00010000:2.0:1789760778.780931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.780934:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.780937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.780940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.780943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.780949:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.780954:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.780970:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760778.781048:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27123712<>27127808 00000100:00100000:3.0:1789760778.781353:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760778.783944:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832960:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760778.831128:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973941 gen 3 00000008:00010000:2.0:1789760778.831164:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760778.831170:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.831179:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.831201:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760778.831225:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493832960:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.831378:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6622, 6622] 00000080:00200000:1.0:1789760778.831391:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.831426:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.831439:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27127808 i_blocks 53255 00000020:00200000:1.0:1789760778.831442:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.831452:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.831480:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27127808, count: 4096 00000080:00200000:1.0:1789760778.831486:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27127808, bytes: 4096 00000080:00200000:1.0:1789760778.831494:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.831501:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.831504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27127808, 27131904) 0 0 00000080:00200000:1.0:1789760778.831507:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.831520:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6623, 6623] 00020000:00200000:1.0:1789760778.831527:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27127808 chunk: [27127808, 27131904) 27131904, 4096 00020000:00200000:1.0:1789760778.831589:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.831607:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6623, 6623] 00000008:00010000:1.0:1789760778.831621:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.831626:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.831635:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27127808, 27131904) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.831645:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27127808=0x19df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.831689:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.831691:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27131904 00000020:00010000:1.0:1789760778.831698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.831701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.831704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.831707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.831710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.831718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.831724:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.831742:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760778.831822:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27127808<>27131904 00000100:00100000:0.0:1789760778.832157:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.832290:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833088:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760778.884014:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760778.884027:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.884036:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.884069:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760778.884155:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833088:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760778.884938:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6623, 6623] 00000080:00200000:1.0:1789760778.884954:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.884959:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.884971:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27131904 i_blocks 53255 00000020:00200000:1.0:1789760778.884974:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.884984:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.885014:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27131904, count: 4096 00000080:00200000:1.0:1789760778.885020:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27131904, bytes: 4096 00000080:00200000:1.0:1789760778.885273:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.885288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.885291:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27131904, 27136000) 0 0 00000080:00200000:1.0:1789760778.885295:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.885310:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6624, 6624] 00020000:00200000:1.0:1789760778.885317:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27131904 chunk: [27131904, 27136000) 27136000, 4096 00020000:00200000:1.0:1789760778.885321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.885339:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6624, 6624] 00000008:00010000:1.0:1789760778.885353:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.885358:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760778.885367:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27131904, 27136000) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.885378:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27131904=0x19e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.885464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.885467:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27136000 00000020:00010000:1.0:1789760778.885474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.885477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.885480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.885483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.885486:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.885506:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.885512:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.885529:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760778.885625:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 27131904<>27136000 00000100:00100000:1.0:1789760778.885675:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.885692:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833216:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760778.939773:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973943 gen 3 00000008:00010000:1.0:1789760778.939824:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:1.0:1789760778.939831:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.939839:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.941995:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:1.0:1789760778.942033:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833216:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760778.945084:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6624, 6624] 00000080:00200000:2.0:1789760778.945105:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.945111:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.945127:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27136000 i_blocks 53255 00000020:00200000:2.0:1789760778.945131:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.945144:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.945171:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27136000, count: 4096 00000080:00200000:2.0:1789760778.945178:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27136000, bytes: 4096 00000080:00200000:2.0:1789760778.945191:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.945199:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.945201:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27136000, 27140096) 0 0 00000080:00200000:2.0:1789760778.945205:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.945217:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6625, 6625] 00020000:00200000:2.0:1789760778.945224:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27136000 chunk: [27136000, 27140096) 27140096, 4096 00020000:00200000:2.0:1789760778.945227:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.945244:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6625, 6625] 00000008:00010000:2.0:1789760778.945259:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.945264:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760778.945274:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27136000, 27140096) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.945284:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27136000=0x19e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.945341:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.945344:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27140096 00000020:00010000:2.0:1789760778.945351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.945355:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.945357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.945372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.945375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.945382:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.945388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.945449:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760778.945528:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27136000<>27140096 00000100:00100000:2.0:1789760778.945586:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.945606:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493833344:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760779.010286:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973944 gen 3 00000008:00010000:2.0:1789760779.010323:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760779.010330:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760779.010339:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760779.010370:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760779.010417:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493833344:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.010571:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6625, 6625] 00000080:00200000:1.0:1789760779.010584:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.010589:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.010602:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27140096 i_blocks 53255 00000020:00200000:1.0:1789760779.010605:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.010616:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.010719:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27140096, count: 4096 00000080:00200000:1.0:1789760779.010727:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27140096, bytes: 4096 00000080:00200000:1.0:1789760779.010737:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.010744:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.010746:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27140096, 27144192) 0 0 00000080:00200000:1.0:1789760779.010751:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.010764:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6626, 6626] 00020000:00200000:1.0:1789760779.010771:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27140096 chunk: [27140096, 27144192) 27144192, 4096 00020000:00200000:1.0:1789760779.010775:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.010791:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6626, 6626] 00000008:00010000:1.0:1789760779.010805:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.010810:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.010820:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27140096, 27144192) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.010834:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27140096=0x19e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.010879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.010882:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27144192 00000020:00010000:1.0:1789760779.010889:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.010893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.010895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.010898:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.010901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.010909:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.010915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.010930:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760779.011723:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 27140096<>27144192 00000100:00100000:0.0:1789760779.011801:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760779.011820:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833472:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.055044:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973945 gen 3 00000008:00010000:0.0:1789760779.055093:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760779.055100:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.055108:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.055133:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:0.0:1789760779.055159:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833472:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.055272:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6626, 6626] 00000080:00200000:1.0:1789760779.055284:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.055290:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.055301:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27144192 i_blocks 53255 00000020:00200000:1.0:1789760779.055304:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.055312:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.055338:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27144192, count: 4096 00000080:00200000:1.0:1789760779.055345:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27144192, bytes: 4096 00000080:00200000:1.0:1789760779.055352:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.055359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.055361:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27144192, 27148288) 0 0 00000080:00200000:1.0:1789760779.055442:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.055618:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6627, 6627] 00020000:00200000:1.0:1789760779.055701:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27144192 chunk: [27144192, 27148288) 27148288, 4096 00020000:00200000:1.0:1789760779.055705:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.055722:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6627, 6627] 00000008:00010000:1.0:1789760779.055736:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.055740:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.055750:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27144192, 27148288) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.055761:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27144192=0x19e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.055805:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.055807:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27148288 00000020:00010000:1.0:1789760779.055814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.055817:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.055819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.055822:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.055825:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.055833:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.055839:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.055855:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760779.055928:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27144192<>27148288 00000100:00100000:1.0:1789760779.055969:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760779.055983:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833600:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760779.088741:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:1.0:1789760779.088751:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.088759:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.088782:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:1.0:1789760779.088806:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833600:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760779.089110:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6627, 6627] 00000080:00200000:2.0:1789760779.089124:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.089129:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.089141:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27148288 i_blocks 53255 00000020:00200000:2.0:1789760779.089144:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.089153:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.089180:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27148288, count: 4096 00000080:00200000:2.0:1789760779.089187:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27148288, bytes: 4096 00000080:00200000:2.0:1789760779.089196:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.089203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.089205:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27148288, 27152384) 0 0 00000080:00200000:2.0:1789760779.089209:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.089223:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6628, 6628] 00020000:00200000:2.0:1789760779.089229:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27148288 chunk: [27148288, 27152384) 27152384, 4096 00020000:00200000:2.0:1789760779.089233:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.089247:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6628, 6628] 00000008:00010000:2.0:1789760779.089259:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.089264:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760779.089274:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27148288, 27152384) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.089285:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27148288=0x19e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.089330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.089332:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27152384 00000020:00010000:2.0:1789760779.089339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.089342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.089345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.089348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.089351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.089356:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.089361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.089376:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760779.089707:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27148288<>27152384 00000100:00100000:3.0:1789760779.089903:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.089921:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493833728:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760779.121379:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973947 gen 3 00000008:00010000:2.0:1789760779.121544:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760779.121551:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760779.121560:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760779.121583:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760779.121609:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493833728:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.121718:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6628, 6628] 00000080:00200000:1.0:1789760779.121731:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.121735:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.121747:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27152384 i_blocks 53255 00000020:00200000:1.0:1789760779.121750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.121758:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.121787:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27152384, count: 4096 00000080:00200000:1.0:1789760779.121793:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27152384, bytes: 4096 00000080:00200000:1.0:1789760779.121802:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.121810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.121812:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27152384, 27156480) 0 0 00000080:00200000:1.0:1789760779.121816:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.121830:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6629, 6629] 00020000:00200000:1.0:1789760779.121845:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27152384 chunk: [27152384, 27156480) 27156480, 4096 00020000:00200000:1.0:1789760779.121849:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.121865:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6629, 6629] 00000008:00010000:1.0:1789760779.121877:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.121882:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.121890:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27152384, 27156480) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.121902:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27152384=0x19e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.121948:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.121951:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27156480 00000020:00010000:1.0:1789760779.121957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.121961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.121963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.121966:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.121969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.121976:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.121982:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.121996:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760779.122081:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 27152384<>27156480 00000100:00100000:0.0:1789760779.122147:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833856:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.158810:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973948 gen 3 00000008:00010000:0.0:1789760779.158906:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760779.158912:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.158920:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.158938:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760779.158963:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493833856:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.159425:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6629, 6629] 00000080:00200000:1.0:1789760779.159438:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.159443:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.159454:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27156480 i_blocks 53255 00000020:00200000:1.0:1789760779.159457:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.159465:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.159489:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27156480, count: 4096 00000080:00200000:1.0:1789760779.159495:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27156480, bytes: 4096 00000080:00200000:1.0:1789760779.159504:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.159511:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.159513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27156480, 27160576) 0 0 00000080:00200000:1.0:1789760779.159517:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.159812:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6630, 6630] 00020000:00200000:1.0:1789760779.159820:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27156480 chunk: [27156480, 27160576) 27160576, 4096 00020000:00200000:1.0:1789760779.159824:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.159908:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6630, 6630] 00000008:00010000:1.0:1789760779.159921:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.159925:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.159934:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27156480, 27160576) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.159945:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27156480=0x19e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.159997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.160000:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27160576 00000020:00010000:1.0:1789760779.160007:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.160010:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.160012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.160016:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.160018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.160033:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.160038:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.160520:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760779.160788:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 27156480<>27160576 00000100:00100000:1.0:1789760779.160828:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833984:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760779.216844:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973949 gen 3 00000008:00010000:1.0:1789760779.216885:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:1.0:1789760779.216892:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.216900:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.217123:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6630, 6630] 00000080:00200000:0.0:1789760779.217136:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.217141:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.217152:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27160576 i_blocks 53255 00000020:00200000:0.0:1789760779.217155:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.217163:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.217188:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27160576, count: 4096 00000080:00200000:0.0:1789760779.217196:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27160576, bytes: 4096 00000080:00200000:0.0:1789760779.217208:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.217215:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.217217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27160576, 27164672) 0 0 00000080:00200000:0.0:1789760779.217221:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.217235:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6631, 6631] 00020000:00200000:0.0:1789760779.217242:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27160576 chunk: [27160576, 27164672) 27164672, 4096 00020000:00200000:0.0:1789760779.217246:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.217261:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6631, 6631] 00000008:00010000:0.0:1789760779.217275:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.217280:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760779.217289:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27160576, 27164672) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.217300:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27160576=0x19e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.217353:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.217356:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27164672 00000020:00010000:0.0:1789760779.217363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.217366:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.217369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.217372:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.217374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.217380:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.217386:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.217450:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760779.217526:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 27160576<>27164672 00000100:00100000:0.0:1789760779.217588:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493834112:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760779.218342:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:1.0:1789760779.218370:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493833984:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.259165:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973950 gen 3 00000008:00010000:0.0:1789760779.259201:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760779.259207:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.259215:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.259274:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760779.259300:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493834112:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760779.259548:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6631, 6631] 00000080:00200000:2.0:1789760779.259561:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.259566:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.259578:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27164672 i_blocks 53255 00000020:00200000:2.0:1789760779.259581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.259591:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.259616:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27164672, count: 4096 00000080:00200000:2.0:1789760779.259621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27164672, bytes: 4096 00000080:00200000:2.0:1789760779.259629:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.259637:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.259638:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27164672, 27168768) 0 0 00000080:00200000:2.0:1789760779.259642:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.259656:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6632, 6632] 00020000:00200000:2.0:1789760779.259662:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27164672 chunk: [27164672, 27168768) 27168768, 4096 00020000:00200000:2.0:1789760779.259666:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.259679:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6632, 6632] 00000008:00010000:2.0:1789760779.259692:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.259696:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760779.259705:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27164672, 27168768) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.259716:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27164672=0x19e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.259758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.259760:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27168768 00000020:00010000:2.0:1789760779.259766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.259769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.259772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.259775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.259778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.259856:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.259862:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.259930:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760779.260005:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27164672<>27168768 00000100:00100000:3.0:1789760779.260156:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760779.260173:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834240:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760779.311677:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973951 gen 3 00000008:00010000:3.0:1789760779.311712:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760779.311718:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.311727:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.311880:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760779.311909:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834240:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760779.314282:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6632, 6632] 00000080:00200000:2.0:1789760779.314302:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.314308:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.314323:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27168768 i_blocks 53255 00000020:00200000:2.0:1789760779.314328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.314342:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.314368:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27168768, count: 4096 00000080:00200000:2.0:1789760779.314374:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27168768, bytes: 4096 00000080:00200000:2.0:1789760779.314387:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.314426:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.314428:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27168768, 27172864) 0 0 00000080:00200000:2.0:1789760779.314432:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.314444:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6633, 6633] 00020000:00200000:2.0:1789760779.314450:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27168768 chunk: [27168768, 27172864) 27172864, 4096 00020000:00200000:2.0:1789760779.314453:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.314470:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6633, 6633] 00000008:00010000:2.0:1789760779.314483:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.314488:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760779.314509:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27168768, 27172864) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.314521:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27168768=0x19e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.314584:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.314587:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27172864 00000020:00010000:2.0:1789760779.314594:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.314598:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.314601:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.314604:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.314607:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.314613:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.314618:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.314653:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760779.314737:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27168768<>27172864 00000100:00100000:3.0:1789760779.314963:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760779.314984:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493834368:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760779.349834:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760779.349845:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.349854:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.349921:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760779.349946:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493834368:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760779.350222:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6633, 6633] 00000080:00200000:2.0:1789760779.350233:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.350238:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.350250:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27172864 i_blocks 53255 00000020:00200000:2.0:1789760779.350254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.350263:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.350290:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27172864, count: 4096 00000080:00200000:2.0:1789760779.350296:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27172864, bytes: 4096 00000080:00200000:2.0:1789760779.350305:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.350312:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.350314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27172864, 27176960) 0 0 00000080:00200000:2.0:1789760779.350318:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.350329:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6634, 6634] 00020000:00200000:2.0:1789760779.350336:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27172864 chunk: [27172864, 27176960) 27176960, 4096 00020000:00200000:2.0:1789760779.350340:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.350354:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6634, 6634] 00000008:00010000:2.0:1789760779.350367:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.350372:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760779.350381:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27172864, 27176960) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.350391:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27172864=0x19ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.350455:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.350458:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27176960 00000020:00010000:2.0:1789760779.350464:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.350467:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.350470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.350473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.350476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.350483:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.350488:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.350503:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760779.350692:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 27172864<>27176960 00000100:00100000:3.0:1789760779.350739:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493834496:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760779.394600:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973952 gen 3 00000100:00100000:3.0:1789760779.394631:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80000 x1876699493834496/t4294973953(4294973953) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760795 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760779.394661:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760779.394666:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.394673:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.394686:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760779.394706:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493834496:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760779.394709:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6634, 6634] 00000080:00200000:2.0:1789760779.394721:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.394726:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.394736:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27176960 i_blocks 53255 00000020:00200000:2.0:1789760779.394740:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.394749:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.394775:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27176960, count: 4096 00000080:00200000:2.0:1789760779.394781:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27176960, bytes: 4096 00000080:00200000:2.0:1789760779.394788:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.394795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.394797:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27176960, 27181056) 0 0 00000080:00200000:2.0:1789760779.394800:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.394812:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6635, 6635] 00020000:00200000:2.0:1789760779.394817:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27176960 chunk: [27176960, 27181056) 27181056, 4096 00020000:00200000:2.0:1789760779.394821:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.394835:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6635, 6635] 00000008:00010000:2.0:1789760779.394849:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.394853:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760779.394862:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27176960, 27181056) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.394873:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27176960=0x19eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.394911:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.394913:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27181056 00000020:00010000:2.0:1789760779.394918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.394921:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.394923:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.394925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.394927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.394931:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.394936:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.394946:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760779.395015:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 27176960<>27181056 00000100:00100000:3.0:1789760779.395202:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834624:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760779.433008:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973954 gen 3 00000008:00010000:3.0:1789760779.433065:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760779.433072:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.433286:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.433374:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760779.433436:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834624:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760779.433832:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6635, 6635] 00000080:00200000:0.0:1789760779.433844:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.433850:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.433861:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27181056 i_blocks 53255 00000020:00200000:0.0:1789760779.433865:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.433874:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.433903:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27181056, count: 4096 00000080:00200000:0.0:1789760779.433908:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27181056, bytes: 4096 00000080:00200000:0.0:1789760779.433917:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.433925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.433927:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27181056, 27185152) 0 0 00000080:00200000:0.0:1789760779.433931:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.433944:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6636, 6636] 00020000:00200000:0.0:1789760779.433951:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27181056 chunk: [27181056, 27185152) 27185152, 4096 00020000:00200000:0.0:1789760779.433955:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.433969:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6636, 6636] 00000008:00010000:0.0:1789760779.433983:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.433987:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760779.433996:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27181056, 27185152) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.434009:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27181056=0x19ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.434066:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.434069:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27185152 00000020:00010000:0.0:1789760779.434076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.434303:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.434308:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.434311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac3f8 (ffff8eb6450b4878/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.434314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4878/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.434321:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.434327:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.434346:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760779.434511:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 27181056<>27185152 00000100:00100000:1.0:1789760779.434699:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493834752:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760779.492908:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973955 gen 3 00000008:00010000:1.0:1789760779.492946:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760779.492953:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.492962:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.492992:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760779.493021:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493834752:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760779.493086:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6636, 6636] 00000080:00200000:3.0:1789760779.493162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.493167:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.493179:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27185152 i_blocks 53255 00000020:00200000:3.0:1789760779.493183:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.493192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.493241:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27185152, count: 4096 00000080:00200000:3.0:1789760779.493248:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27185152, bytes: 4096 00000080:00200000:3.0:1789760779.493261:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.493269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760779.493271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27185152, 27189248) 0 0 00000080:00200000:3.0:1789760779.493274:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.493287:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6637, 6637] 00020000:00200000:3.0:1789760779.493294:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27185152 chunk: [27185152, 27189248) 27189248, 4096 00020000:00200000:3.0:1789760779.493298:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.493315:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6637, 6637] 00000008:00010000:3.0:1789760779.493330:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.493336:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760779.493346:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27185152, 27189248) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.493356:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27185152=0x19ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.493431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.493434:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27189248 00000020:00010000:3.0:1789760779.493441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.493445:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.493447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.493450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.493453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.493459:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.493464:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.493479:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760779.493691:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27185152<>27189248 00000100:00100000:3.0:1789760779.493741:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760779.493756:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834880:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760779.501270:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760779.540128:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973956 gen 3 00000008:00010000:3.0:1789760779.540166:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760779.540173:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.540183:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.540211:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760779.540237:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493834880:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.540346:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6637, 6637] 00000080:00200000:1.0:1789760779.540360:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.540365:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.540378:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27189248 i_blocks 53255 00000020:00200000:1.0:1789760779.540381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.540390:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.540450:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27189248, count: 4096 00000080:00200000:1.0:1789760779.540456:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27189248, bytes: 4096 00000080:00200000:1.0:1789760779.540465:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.540473:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.540476:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27189248, 27193344) 0 0 00000080:00200000:1.0:1789760779.540480:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.540493:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6638, 6638] 00020000:00200000:1.0:1789760779.540501:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27189248 chunk: [27189248, 27193344) 27193344, 4096 00020000:00200000:1.0:1789760779.540506:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.540520:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6638, 6638] 00000008:00010000:1.0:1789760779.540546:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.540552:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.540560:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27189248, 27193344) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.540571:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27189248=0x19ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.540617:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.540620:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27193344 00000020:00010000:1.0:1789760779.540627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.540630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.540633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.540637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.540639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.540659:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.540665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.540680:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760779.540761:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 27189248<>27193344 00000100:00100000:1.0:1789760779.540805:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.540818:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835008:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760779.581645:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973957 gen 3 00000008:00010000:1.0:1789760779.581686:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:1.0:1789760779.581694:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.581703:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.581728:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:1.0:1789760779.581757:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835008:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760779.581860:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6638, 6638] 00000080:00200000:3.0:1789760779.581873:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760779.581879:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760779.581890:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27193344 i_blocks 53255 00000020:00200000:3.0:1789760779.581893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760779.581902:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760779.581931:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27193344, count: 4096 00000080:00200000:3.0:1789760779.581937:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27193344, bytes: 4096 00000080:00200000:3.0:1789760779.581946:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760779.581953:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760779.581955:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27193344, 27197440) 0 0 00000080:00200000:3.0:1789760779.581959:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760779.581970:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6639, 6639] 00020000:00200000:3.0:1789760779.581977:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27193344 chunk: [27193344, 27197440) 27197440, 4096 00020000:00200000:3.0:1789760779.581981:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760779.581996:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6639, 6639] 00000008:00010000:3.0:1789760779.582010:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760779.582014:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760779.582023:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27193344, 27197440) append 0 append lockpos 0 00000080:00200000:3.0:1789760779.582034:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27193344=0x19ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760779.582088:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760779.582091:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27197440 00000020:00010000:3.0:1789760779.582097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760779.582111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.582114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760779.582117:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760779.582120:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760779.582126:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760779.582131:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760779.582145:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760779.582222:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27193344<>27197440 00000100:00100000:2.0:1789760779.582426:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760779.582441:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493835136:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760779.645799:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973958 gen 3 00000008:00010000:3.0:1789760779.645851:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760779.645859:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760779.645869:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760779.645890:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760779.645919:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493835136:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.646003:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6639, 6639] 00000080:00200000:1.0:1789760779.646023:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.646029:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.646043:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27197440 i_blocks 53255 00000020:00200000:1.0:1789760779.646058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.646072:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.646100:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27197440, count: 4096 00000080:00200000:1.0:1789760779.646107:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27197440, bytes: 4096 00000080:00200000:1.0:1789760779.646119:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.646125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.646127:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27197440, 27201536) 0 0 00000080:00200000:1.0:1789760779.646131:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.646145:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6640, 6640] 00020000:00200000:1.0:1789760779.646151:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27197440 chunk: [27197440, 27201536) 27201536, 4096 00020000:00200000:1.0:1789760779.646155:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.646170:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6640, 6640] 00000008:00010000:1.0:1789760779.646185:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.646189:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.646198:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27197440, 27201536) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.646207:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27197440=0x19f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.646259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.646262:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27201536 00000020:00010000:1.0:1789760779.646269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.646272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.646275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.646278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.646280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.646289:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.646295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.646313:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760779.647014:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 27197440<>27201536 00000100:00100000:0.0:1789760779.649170:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.649201:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835264:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760779.752882:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760779.752894:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.752903:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.752923:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760779.752950:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835264:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.753222:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6640, 6640] 00000080:00200000:1.0:1789760779.753234:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.753240:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.753252:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27201536 i_blocks 53255 00000020:00200000:1.0:1789760779.753255:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.753264:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.753292:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27201536, count: 4096 00000080:00200000:1.0:1789760779.753298:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27201536, bytes: 4096 00000080:00200000:1.0:1789760779.753307:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.753314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.753316:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27201536, 27205632) 0 0 00000080:00200000:1.0:1789760779.753320:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.753333:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6641, 6641] 00020000:00200000:1.0:1789760779.753340:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27201536 chunk: [27201536, 27205632) 27205632, 4096 00020000:00200000:1.0:1789760779.753344:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.753358:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6641, 6641] 00000008:00010000:1.0:1789760779.753372:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.753376:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.753386:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27201536, 27205632) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.753442:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27201536=0x19f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.753482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.753485:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27205632 00000020:00010000:1.0:1789760779.753491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.753494:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.753505:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.753508:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.753510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.753518:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.753612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.753628:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92840 00000008:00100000:1.0:1789760779.753707:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 27201536<>27205632 00000100:00100000:0.0:1789760779.754053:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835392:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.806169:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973960 gen 3 00000008:00010000:0.0:1789760779.806215:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92840 00000020:00200000:0.0:1789760779.806222:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.806230:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.806250:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92840 00000100:00100000:0.0:1789760779.806276:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835392:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.808995:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6641, 6641] 00000080:00200000:1.0:1789760779.809019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.809025:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.809040:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27205632 i_blocks 53255 00000020:00200000:1.0:1789760779.809054:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.809067:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.809097:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27205632, count: 4096 00000080:00200000:1.0:1789760779.809103:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27205632, bytes: 4096 00000080:00200000:1.0:1789760779.809115:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.809122:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.809124:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27205632, 27209728) 0 0 00000080:00200000:1.0:1789760779.809128:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.809140:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6642, 6642] 00020000:00200000:1.0:1789760779.809147:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27205632 chunk: [27205632, 27209728) 27209728, 4096 00020000:00200000:1.0:1789760779.809151:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.809168:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6642, 6642] 00000008:00010000:1.0:1789760779.809183:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.809188:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.809198:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27205632, 27209728) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.809211:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27205632=0x19f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.809275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.809278:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27209728 00000020:00010000:1.0:1789760779.809285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.809289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.809292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.809295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.809297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.809308:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.809314:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.809333:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760779.809464:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 27205632<>27209728 00000100:00100000:0.0:1789760779.809778:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.809802:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835520:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.865636:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973961 gen 3 00000008:00010000:0.0:1789760779.865676:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760779.865682:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.865690:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.865709:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760779.865735:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835520:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.865936:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6642, 6642] 00000080:00200000:1.0:1789760779.865949:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.865954:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.866033:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27209728 i_blocks 53255 00000020:00200000:1.0:1789760779.866037:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.866055:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.866081:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27209728, count: 4096 00000080:00200000:1.0:1789760779.866086:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27209728, bytes: 4096 00000080:00200000:1.0:1789760779.866094:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.866108:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.866110:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27209728, 27213824) 0 0 00000080:00200000:1.0:1789760779.866114:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.866127:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6643, 6643] 00020000:00200000:1.0:1789760779.866133:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27209728 chunk: [27209728, 27213824) 27213824, 4096 00020000:00200000:1.0:1789760779.866137:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.866151:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6643, 6643] 00000008:00010000:1.0:1789760779.866163:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.866168:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.866176:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27209728, 27213824) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.866187:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27209728=0x19f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.866227:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.866229:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27213824 00000020:00010000:1.0:1789760779.866235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.866238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.866241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.866244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.866247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.866254:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.866259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.866273:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760779.866345:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 27209728<>27213824 00000100:00100000:0.0:1789760779.866499:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760779.866513:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835648:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.914988:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973962 gen 3 00000008:00010000:0.0:1789760779.915030:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760779.915037:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.915055:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.915078:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760779.915115:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835648:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.915383:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6643, 6643] 00000080:00200000:1.0:1789760779.915429:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.915436:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.915448:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27213824 i_blocks 53255 00000020:00200000:1.0:1789760779.915452:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.915461:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.915487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27213824, count: 4096 00000080:00200000:1.0:1789760779.915493:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27213824, bytes: 4096 00000080:00200000:1.0:1789760779.915504:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.915511:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.915513:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27213824, 27217920) 0 0 00000080:00200000:1.0:1789760779.915517:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.915594:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6644, 6644] 00020000:00200000:1.0:1789760779.915600:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27213824 chunk: [27213824, 27217920) 27217920, 4096 00020000:00200000:1.0:1789760779.915604:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.915620:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6644, 6644] 00000008:00010000:1.0:1789760779.915634:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.915639:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.915648:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27213824, 27217920) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.915659:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27213824=0x19f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.915704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.915706:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27217920 00000020:00010000:1.0:1789760779.915713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.915716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.915719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.915722:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.915725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.915733:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.915740:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.915755:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760779.915910:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 27213824<>27217920 00000100:00100000:0.0:1789760779.916006:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.916021:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835776:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.955441:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973963 gen 3 00000008:00010000:0.0:1789760779.955477:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760779.955483:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.955492:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.955510:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760779.955544:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493835776:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.955921:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6644, 6644] 00000080:00200000:1.0:1789760779.955936:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.955941:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.955953:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27217920 i_blocks 53255 00000020:00200000:1.0:1789760779.955956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.955964:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.955994:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27217920, count: 4096 00000080:00200000:1.0:1789760779.956000:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27217920, bytes: 4096 00000080:00200000:1.0:1789760779.956009:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.956017:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.956019:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27217920, 27222016) 0 0 00000080:00200000:1.0:1789760779.956024:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.956038:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6645, 6645] 00020000:00200000:1.0:1789760779.956054:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27217920 chunk: [27217920, 27222016) 27222016, 4096 00020000:00200000:1.0:1789760779.956058:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.956073:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6645, 6645] 00000008:00010000:1.0:1789760779.956088:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.956094:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.956113:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27217920, 27222016) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.956125:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27217920=0x19f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.956172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.956175:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27222016 00000020:00010000:1.0:1789760779.956182:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.956186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.956189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.956193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.956196:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.956205:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.956211:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.956225:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760779.956311:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 27217920<>27222016 00000100:00100000:0.0:1789760779.956460:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760779.956478:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835904:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760779.993490:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973964 gen 3 00000008:00010000:0.0:1789760779.993634:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760779.993640:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.993649:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.993789:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760779.993817:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493835904:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760779.993892:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6645, 6645] 00000080:00200000:1.0:1789760779.993905:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.993910:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.993922:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27222016 i_blocks 53255 00000020:00200000:1.0:1789760779.993925:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.993932:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.994024:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27222016, count: 4096 00000080:00200000:1.0:1789760779.994031:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27222016, bytes: 4096 00000080:00200000:1.0:1789760779.994040:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.994056:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.994058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27222016, 27226112) 0 0 00000080:00200000:1.0:1789760779.994061:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.994075:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6646, 6646] 00020000:00200000:1.0:1789760779.994083:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27222016 chunk: [27222016, 27226112) 27226112, 4096 00020000:00200000:1.0:1789760779.994086:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.994159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6646, 6646] 00000008:00010000:1.0:1789760779.994174:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.994178:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760779.994186:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27222016, 27226112) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.994198:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27222016=0x19f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.994242:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.994244:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27226112 00000020:00010000:1.0:1789760779.994252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.994255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.994257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.994260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.994263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.994272:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.994278:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.994292:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e920c0 00000008:00100000:1.0:1789760779.994369:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 27222016<>27226112 00000100:00100000:0.0:1789760779.994431:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493836032:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.024121:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973965 gen 3 00000008:00010000:0.0:1789760780.024155:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e920c0 00000020:00200000:0.0:1789760780.024161:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.024170:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.024192:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e920c0 00000100:00100000:0.0:1789760780.024217:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493836032:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.024696:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6646, 6646] 00000080:00200000:1.0:1789760780.024710:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.024718:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.024730:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27226112 i_blocks 53255 00000020:00200000:1.0:1789760780.024733:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.024742:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.024781:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27226112, count: 4096 00000080:00200000:1.0:1789760780.024786:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27226112, bytes: 4096 00000080:00200000:1.0:1789760780.024796:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.024802:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.024804:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27226112, 27230208) 0 0 00000080:00200000:1.0:1789760780.024808:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.024821:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6647, 6647] 00020000:00200000:1.0:1789760780.024827:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27226112 chunk: [27226112, 27230208) 27230208, 4096 00020000:00200000:1.0:1789760780.024830:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.024846:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6647, 6647] 00000008:00010000:1.0:1789760780.024858:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.024863:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.024882:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27226112, 27230208) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.024894:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27226112=0x19f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.024937:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.024940:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27230208 00000020:00010000:1.0:1789760780.024946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.024949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.024952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.024955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.024957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.024965:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.024970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.024986:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760780.025080:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 27226112<>27230208 00000100:00100000:0.0:1789760780.025201:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836160:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.064334:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973966 gen 3 00000008:00010000:0.0:1789760780.064369:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760780.064376:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.064384:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.064432:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760780.064458:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836160:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.064563:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6647, 6647] 00000080:00200000:1.0:1789760780.064576:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.064580:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.064592:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27230208 i_blocks 53255 00000020:00200000:1.0:1789760780.064596:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.064606:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.064631:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27230208, count: 4096 00000080:00200000:1.0:1789760780.064636:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27230208, bytes: 4096 00000080:00200000:1.0:1789760780.064643:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.064650:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.064652:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27230208, 27234304) 0 0 00000080:00200000:1.0:1789760780.064656:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.064669:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6648, 6648] 00020000:00200000:1.0:1789760780.064676:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27230208 chunk: [27230208, 27234304) 27234304, 4096 00020000:00200000:1.0:1789760780.064680:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.064694:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6648, 6648] 00000008:00010000:1.0:1789760780.064707:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.064712:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.064720:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27230208, 27234304) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.064729:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27230208=0x19f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.064771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.064774:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27234304 00000020:00010000:1.0:1789760780.064780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.064783:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.064786:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.064789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.064792:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.064800:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.064806:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.064820:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760780.064895:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 27230208<>27234304 00000100:00100000:0.0:1789760780.064955:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493836288:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.111129:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973967 gen 3 00000008:00010000:0.0:1789760780.111165:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760780.111171:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.111178:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.111198:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760780.111224:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493836288:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.111329:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6648, 6648] 00000080:00200000:1.0:1789760780.111341:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.111346:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.111357:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27234304 i_blocks 53255 00000020:00200000:1.0:1789760780.111360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.111368:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.111391:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27234304, count: 4096 00000080:00200000:1.0:1789760780.111427:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27234304, bytes: 4096 00000080:00200000:1.0:1789760780.111435:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.111442:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.111443:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27234304, 27238400) 0 0 00000080:00200000:1.0:1789760780.111448:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.111461:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6649, 6649] 00020000:00200000:1.0:1789760780.111468:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27234304 chunk: [27234304, 27238400) 27238400, 4096 00020000:00200000:1.0:1789760780.111471:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.111485:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6649, 6649] 00000008:00010000:1.0:1789760780.111498:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.111502:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.111510:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27234304, 27238400) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.111520:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27234304=0x19f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.111558:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.111560:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27238400 00000020:00010000:1.0:1789760780.111565:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.111568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.111571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.111574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.111576:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.111583:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.111589:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.111602:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760780.111675:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 27234304<>27238400 00000100:00100000:0.0:1789760780.111827:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836416:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.174767:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973968 gen 3 00000008:00010000:0.0:1789760780.174806:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760780.174812:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.174820:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.174842:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760780.174869:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836416:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.175030:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6649, 6649] 00000080:00200000:1.0:1789760780.175055:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.175060:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.175072:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27238400 i_blocks 53255 00000020:00200000:1.0:1789760780.175075:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.175083:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.175117:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27238400, count: 4096 00000080:00200000:1.0:1789760780.175124:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27238400, bytes: 4096 00000080:00200000:1.0:1789760780.175131:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.175145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.175147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27238400, 27242496) 0 0 00000080:00200000:1.0:1789760780.175151:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.175164:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6650, 6650] 00020000:00200000:1.0:1789760780.175171:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27238400 chunk: [27238400, 27242496) 27242496, 4096 00020000:00200000:1.0:1789760780.175175:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.175192:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6650, 6650] 00000008:00010000:1.0:1789760780.175205:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.175209:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.175218:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27238400, 27242496) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.175227:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27238400=0x19fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.175282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.175285:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27242496 00000020:00010000:1.0:1789760780.175292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.175295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.175302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.175306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.175309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.175317:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.175322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.175336:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760780.175459:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 27238400<>27242496 00000100:00100000:1.0:1789760780.175489:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760780.175501:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836544:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.219710:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973969 gen 3 00000008:00010000:1.0:1789760780.219744:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760780.219751:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.219759:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.219828:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760780.219855:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836544:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760780.220911:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6650, 6650] 00000080:00200000:3.0:1789760780.220926:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.220932:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.220944:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27242496 i_blocks 53255 00000020:00200000:3.0:1789760780.220948:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.220956:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.220982:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27242496, count: 4096 00000080:00200000:3.0:1789760780.220988:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27242496, bytes: 4096 00000080:00200000:3.0:1789760780.221126:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.221136:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.221138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27242496, 27246592) 0 0 00000080:00200000:3.0:1789760780.221143:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.221158:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6651, 6651] 00020000:00200000:3.0:1789760780.221167:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27242496 chunk: [27242496, 27246592) 27246592, 4096 00020000:00200000:3.0:1789760780.221171:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.221190:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6651, 6651] 00000008:00010000:3.0:1789760780.221205:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.221210:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760780.221220:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27242496, 27246592) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.221310:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27242496=0x19fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.221366:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.221369:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27246592 00000020:00010000:3.0:1789760780.221377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.221380:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.221383:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.221386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.221389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.221423:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.221429:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.221526:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760780.221915:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27242496<>27246592 00000100:00100000:2.0:1789760780.221993:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493836672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.266256:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973970 gen 3 00000008:00010000:3.0:1789760780.266302:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760780.266309:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.266317:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.266342:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760780.266371:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493836672:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.266536:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6651, 6651] 00000080:00200000:1.0:1789760780.266549:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.266554:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.266566:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27246592 i_blocks 53255 00000020:00200000:1.0:1789760780.266570:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.266578:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.266605:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27246592, count: 4096 00000080:00200000:1.0:1789760780.266611:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27246592, bytes: 4096 00000080:00200000:1.0:1789760780.266621:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.266628:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.266630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27246592, 27250688) 0 0 00000080:00200000:1.0:1789760780.266634:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.266646:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6652, 6652] 00020000:00200000:1.0:1789760780.266653:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27246592 chunk: [27246592, 27250688) 27250688, 4096 00020000:00200000:1.0:1789760780.266657:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.266673:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6652, 6652] 00000008:00010000:1.0:1789760780.266686:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.266691:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.266699:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27246592, 27250688) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.266709:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27246592=0x19fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.266751:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.266754:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27250688 00000020:00010000:1.0:1789760780.266760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.266764:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.266766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.266769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.266772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.266780:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.266786:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.266800:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760780.266894:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 27246592<>27250688 00000100:00100000:0.0:1789760780.267283:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836800:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.313472:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973971 gen 3 00000008:00010000:0.0:1789760780.313513:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760780.313519:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.314362:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.314388:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760780.314448:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836800:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.314697:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6652, 6652] 00000080:00200000:1.0:1789760780.314710:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.314715:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.314728:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27250688 i_blocks 53255 00000020:00200000:1.0:1789760780.314731:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.314740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.314769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27250688, count: 4096 00000080:00200000:1.0:1789760780.314775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27250688, bytes: 4096 00000080:00200000:1.0:1789760780.314785:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.314792:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.314794:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27250688, 27254784) 0 0 00000080:00200000:1.0:1789760780.314797:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.314810:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6653, 6653] 00020000:00200000:1.0:1789760780.314816:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27250688 chunk: [27250688, 27254784) 27254784, 4096 00020000:00200000:1.0:1789760780.314820:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.314835:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6653, 6653] 00000008:00010000:1.0:1789760780.314848:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.314852:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.314861:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27250688, 27254784) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.314872:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27250688=0x19fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.314917:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.314920:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27254784 00000020:00010000:1.0:1789760780.314925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.314928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.314931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.314934:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.314936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.314944:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.314949:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.314965:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760780.315051:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27250688<>27254784 00000100:00100000:0.0:1789760780.315282:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760780.315358:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836928:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760780.357168:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760780.357178:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.357187:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.357212:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760780.357237:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493836928:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.357291:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6653, 6653] 00000080:00200000:1.0:1789760780.357308:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.357313:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.357324:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27254784 i_blocks 53255 00000020:00200000:1.0:1789760780.357329:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.357338:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.357360:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27254784, count: 4096 00000080:00200000:1.0:1789760780.357365:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27254784, bytes: 4096 00000080:00200000:1.0:1789760780.357373:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.357380:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.357382:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27254784, 27258880) 0 0 00000080:00200000:1.0:1789760780.357386:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.357426:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6654, 6654] 00020000:00200000:1.0:1789760780.357433:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27254784 chunk: [27254784, 27258880) 27258880, 4096 00020000:00200000:1.0:1789760780.357436:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.357451:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6654, 6654] 00000008:00010000:1.0:1789760780.357464:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.357469:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.357478:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27254784, 27258880) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.357489:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27254784=0x19fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.357543:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.357546:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27258880 00000020:00010000:1.0:1789760780.357552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.357555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.357558:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.357561:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.357564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.357571:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.357577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.357589:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760780.357669:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 27254784<>27258880 00000100:00100000:1.0:1789760780.357708:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760780.357720:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837056:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.402452:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973973 gen 3 00000008:00010000:1.0:1789760780.402500:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760780.402507:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.402514:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.402543:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760780.402568:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837056:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760780.402703:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6654, 6654] 00000080:00200000:3.0:1789760780.402714:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.402719:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.402731:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27258880 i_blocks 53255 00000020:00200000:3.0:1789760780.402735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.402743:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.402771:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27258880, count: 4096 00000080:00200000:3.0:1789760780.402777:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27258880, bytes: 4096 00000080:00200000:3.0:1789760780.402786:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.402793:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.402795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27258880, 27262976) 0 0 00000080:00200000:3.0:1789760780.402799:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.402811:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6655, 6655] 00020000:00200000:3.0:1789760780.402828:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27258880 chunk: [27258880, 27262976) 27262976, 4096 00020000:00200000:3.0:1789760780.402832:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.402847:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6655, 6655] 00000008:00010000:3.0:1789760780.402861:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.402865:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760780.402874:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27258880, 27262976) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.402886:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27258880=0x19ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.402928:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.402930:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27262976 00000020:00010000:3.0:1789760780.402936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.402940:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.402942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.402945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.402947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.402953:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.402959:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.402972:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760780.403064:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27258880<>27262976 00000100:00100000:3.0:1789760780.403110:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.403124:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837184:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.439424:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973974 gen 3 00000008:00010000:3.0:1789760780.439457:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760780.439463:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.439470:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.439496:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760780.439531:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837184:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.441208:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6655, 6655] 00000080:00200000:1.0:1789760780.441225:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.441232:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.441244:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27262976 i_blocks 53255 00000020:00200000:1.0:1789760780.441249:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.441260:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.441287:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27262976, count: 4096 00000080:00200000:1.0:1789760780.441293:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27262976, bytes: 4096 00000080:00200000:1.0:1789760780.441304:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.441311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.441313:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27262976, 27267072) 0 0 00000080:00200000:1.0:1789760780.441317:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.441329:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6656, 6656] 00020000:00200000:1.0:1789760780.441336:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27262976 chunk: [27262976, 27267072) 27267072, 4096 00020000:00200000:1.0:1789760780.441339:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.441354:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6656, 6656] 00000008:00010000:1.0:1789760780.441368:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.441373:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.441381:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27262976, 27267072) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.441391:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27262976=0x1a00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.441471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.441474:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27267072 00000020:00010000:1.0:1789760780.441480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.441484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.441487:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.441490:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.441493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.441501:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.441506:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.441521:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760780.441608:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 27262976<>27267072 00000100:00100000:0.0:1789760780.442106:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837312:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760780.514232:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973975 gen 3 00000008:00010000:0.0:1789760780.514269:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760780.514276:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.514284:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.514364:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760780.514429:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837312:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.516803:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6656, 6656] 00000080:00200000:1.0:1789760780.516823:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.516828:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.516841:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27267072 i_blocks 53255 00000020:00200000:1.0:1789760780.516844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.516854:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.516893:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27267072, count: 4096 00000080:00200000:1.0:1789760780.516901:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27267072, bytes: 4096 00000080:00200000:1.0:1789760780.516913:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.516921:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.516923:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27267072, 27271168) 0 0 00000080:00200000:1.0:1789760780.516926:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.516939:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6657, 6657] 00020000:00200000:1.0:1789760780.516946:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27267072 chunk: [27267072, 27271168) 27271168, 4096 00020000:00200000:1.0:1789760780.516950:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.516964:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6657, 6657] 00000008:00010000:1.0:1789760780.516978:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.516982:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.516991:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27267072, 27271168) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.517003:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27267072=0x1a01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.518423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.518430:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27271168 00000020:00010000:1.0:1789760780.518442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.518445:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.518450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.518454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.518457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.518467:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.518474:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.518493:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760780.518868:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27267072<>27271168 00000100:00100000:1.0:1789760780.518933:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760780.518954:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837440:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760780.524752:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760780.566431:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973976 gen 3 00000008:00010000:1.0:1789760780.566469:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760780.566476:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.566485:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.566652:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760780.566678:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493837440:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760780.568032:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6657, 6657] 00000080:00200000:3.0:1789760780.568058:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.568064:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.568079:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27271168 i_blocks 55305 00000020:00200000:3.0:1789760780.568084:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.568161:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.568189:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27271168, count: 4096 00000080:00200000:3.0:1789760780.568195:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27271168, bytes: 4096 00000080:00200000:3.0:1789760780.568207:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.568214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.568216:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27271168, 27275264) 0 0 00000080:00200000:3.0:1789760780.568220:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.568234:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6658, 6658] 00020000:00200000:3.0:1789760780.568241:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27271168 chunk: [27271168, 27275264) 27275264, 4096 00020000:00200000:3.0:1789760780.568245:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.568260:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6658, 6658] 00000008:00010000:3.0:1789760780.568275:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.568279:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760780.568289:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27271168, 27275264) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.568301:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27271168=0x1a02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.568355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.568358:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27275264 00000020:00010000:3.0:1789760780.568365:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.568369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.568371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.568375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.568378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.568384:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.568390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.568926:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760780.569010:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27271168<>27275264 00000100:00100000:2.0:1789760780.569073:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760780.570059:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493837568:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.634474:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973977 gen 3 00000008:00010000:3.0:1789760780.634510:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760780.634517:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.634534:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.634555:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760780.634579:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493837568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760780.634914:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6658, 6658] 00000080:00200000:2.0:1789760780.634926:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.634931:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.634943:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27275264 i_blocks 55305 00000020:00200000:2.0:1789760780.634946:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.634955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.634983:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27275264, count: 4096 00000080:00200000:2.0:1789760780.634989:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27275264, bytes: 4096 00000080:00200000:2.0:1789760780.634998:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.635005:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.635007:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27275264, 27279360) 0 0 00000080:00200000:2.0:1789760780.635011:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.635023:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6659, 6659] 00020000:00200000:2.0:1789760780.635030:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27275264 chunk: [27275264, 27279360) 27279360, 4096 00020000:00200000:2.0:1789760780.635033:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.635059:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6659, 6659] 00000008:00010000:2.0:1789760780.635072:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.635077:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760780.635087:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27275264, 27279360) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.635097:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27275264=0x1a03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.635140:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.635143:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27279360 00000020:00010000:2.0:1789760780.635150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.635153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.635155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.635158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.635160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.635167:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.635173:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.635187:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760780.635270:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27275264<>27279360 00000100:00100000:3.0:1789760780.635463:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.635479:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.678671:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973978 gen 3 00000008:00010000:3.0:1789760780.678708:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760780.678715:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.678725:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.678790:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760780.678825:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760780.679028:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6659, 6659] 00000080:00200000:2.0:1789760780.679049:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.679055:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.679067:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27279360 i_blocks 55305 00000020:00200000:2.0:1789760780.679070:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.679079:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.679108:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27279360, count: 4096 00000080:00200000:2.0:1789760780.679113:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27279360, bytes: 4096 00000080:00200000:2.0:1789760780.679122:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.679189:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.679192:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27279360, 27283456) 0 0 00000080:00200000:2.0:1789760780.679196:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.679211:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6660, 6660] 00020000:00200000:2.0:1789760780.679217:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27279360 chunk: [27279360, 27283456) 27283456, 4096 00020000:00200000:2.0:1789760780.679222:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.679238:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6660, 6660] 00000008:00010000:2.0:1789760780.679253:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.679259:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760780.679268:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27279360, 27283456) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.679278:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27279360=0x1a04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.679328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.679330:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27283456 00000020:00010000:2.0:1789760780.679336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.679340:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.679343:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.679346:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.679349:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.679357:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.679362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.679378:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760780.679724:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27279360<>27283456 00000100:00100000:3.0:1789760780.679782:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.725984:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973979 gen 3 00000008:00010000:3.0:1789760780.726019:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760780.726025:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.726034:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.726120:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760780.726145:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493837824:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760780.729190:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6660, 6660] 00000080:00200000:1.0:1789760780.729210:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760780.729216:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760780.729230:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27283456 i_blocks 55305 00000020:00200000:1.0:1789760780.729234:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760780.729247:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760780.729274:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27283456, count: 4096 00000080:00200000:1.0:1789760780.729281:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27283456, bytes: 4096 00000080:00200000:1.0:1789760780.729293:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760780.729300:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760780.729302:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27283456, 27287552) 0 0 00000080:00200000:1.0:1789760780.729306:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760780.729319:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6661, 6661] 00020000:00200000:1.0:1789760780.729326:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27283456 chunk: [27283456, 27287552) 27287552, 4096 00020000:00200000:1.0:1789760780.729331:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760780.729346:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6661, 6661] 00000008:00010000:1.0:1789760780.729360:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760780.729365:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760780.729374:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27283456, 27287552) append 0 append lockpos 0 00000080:00200000:1.0:1789760780.729385:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27283456=0x1a05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760780.729456:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760780.729458:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27287552 00000020:00010000:1.0:1789760780.729465:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760780.729469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.729472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760780.729475:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760780.729477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760780.729486:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760780.729491:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.729588:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760780.729675:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27283456<>27287552 00000100:00100000:1.0:1789760780.729732:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760780.730074:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493837952:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.769771:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973980 gen 3 00000008:00010000:1.0:1789760780.769810:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760780.769816:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.769824:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.769833:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760780.769857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493837952:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760780.769893:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6661, 6661] 00000080:00200000:0.0:1789760780.769906:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.769911:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.769923:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27287552 i_blocks 55305 00000020:00200000:0.0:1789760780.769925:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.769933:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.769955:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27287552, count: 4096 00000080:00200000:0.0:1789760780.769960:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27287552, bytes: 4096 00000080:00200000:0.0:1789760780.769969:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.769975:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.769977:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27287552, 27291648) 0 0 00000080:00200000:0.0:1789760780.769981:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.769994:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6662, 6662] 00020000:00200000:0.0:1789760780.770001:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27287552 chunk: [27287552, 27291648) 27291648, 4096 00020000:00200000:0.0:1789760780.770004:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.770019:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6662, 6662] 00000008:00010000:0.0:1789760780.770032:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.770047:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760780.770059:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27287552, 27291648) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.770070:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27287552=0x1a06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.770183:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.770186:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27291648 00000020:00010000:0.0:1789760780.770194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.770197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.770200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.770203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.770206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.770212:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.770218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.770231:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760780.770310:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 27287552<>27291648 00000100:00100000:1.0:1789760780.770371:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493838080:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.813122:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973981 gen 3 00000008:00010000:1.0:1789760780.813159:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760780.813166:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.813176:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.813197:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760780.813230:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493838080:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760780.813368:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6662, 6662] 00000080:00200000:0.0:1789760780.813380:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.813385:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.813429:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27291648 i_blocks 55305 00000020:00200000:0.0:1789760780.813433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.813445:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.813474:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27291648, count: 4096 00000080:00200000:0.0:1789760780.813483:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27291648, bytes: 4096 00000080:00200000:0.0:1789760780.813497:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.813503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.813505:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27291648, 27295744) 0 0 00000080:00200000:0.0:1789760780.813512:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.813527:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6663, 6663] 00020000:00200000:0.0:1789760780.813533:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27291648 chunk: [27291648, 27295744) 27295744, 4096 00020000:00200000:0.0:1789760780.813537:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.813687:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6663, 6663] 00000008:00010000:0.0:1789760780.813702:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.813710:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760780.813721:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27291648, 27295744) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.813731:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27291648=0x1a07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.813786:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.813789:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27295744 00000020:00010000:0.0:1789760780.813795:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.813802:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.813805:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.813808:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.813814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.813820:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.813825:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.813874:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760780.813967:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 27291648<>27295744 00000100:00100000:1.0:1789760780.814052:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493838208:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.858919:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973982 gen 3 00000008:00010000:1.0:1789760780.858953:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760780.858958:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.858965:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.858979:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000080:00200000:2.0:1789760780.859002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6663, 6663] 00000100:00100000:1.0:1789760780.859004:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493838208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760780.859015:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.859019:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.859031:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27295744 i_blocks 55305 00000020:00200000:2.0:1789760780.859034:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.859054:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.859080:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27295744, count: 4096 00000080:00200000:2.0:1789760780.859086:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27295744, bytes: 4096 00000080:00200000:2.0:1789760780.859095:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.859102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.859104:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27295744, 27299840) 0 0 00000080:00200000:2.0:1789760780.859107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.859120:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6664, 6664] 00020000:00200000:2.0:1789760780.859127:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27295744 chunk: [27295744, 27299840) 27299840, 4096 00020000:00200000:2.0:1789760780.859131:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.859146:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6664, 6664] 00000008:00010000:2.0:1789760780.859160:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.859164:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760780.859176:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27295744, 27299840) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.859186:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27295744=0x1a08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.859236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.859238:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27299840 00000020:00010000:2.0:1789760780.859245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.859248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.859250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.859253:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.859256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.859262:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.859267:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760780.859290:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:1.0:1789760780.859431:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27295744<>27299840 00000100:00100000:1.0:1789760780.859474:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760780.859487:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493838336:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760780.895532:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973983 gen 3 00000008:00010000:1.0:1789760780.895570:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:1.0:1789760780.895576:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760780.895587:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760780.895605:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:1.0:1789760780.895632:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493838336:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760780.895735:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6664, 6664] 00000080:00200000:2.0:1789760780.895748:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.895753:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.895766:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27299840 i_blocks 55305 00000020:00200000:2.0:1789760780.895770:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.895779:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.895806:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27299840, count: 4096 00000080:00200000:2.0:1789760780.895812:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27299840, bytes: 4096 00000080:00200000:2.0:1789760780.895821:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.895828:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.895830:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27299840, 27303936) 0 0 00000080:00200000:2.0:1789760780.895834:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.895857:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6665, 6665] 00020000:00200000:2.0:1789760780.895864:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27299840 chunk: [27299840, 27303936) 27303936, 4096 00020000:00200000:2.0:1789760780.895868:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.895884:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6665, 6665] 00000008:00010000:2.0:1789760780.895897:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.895902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760780.895912:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27299840, 27303936) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.895923:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27299840=0x1a09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.895968:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.895970:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27303936 00000020:00010000:2.0:1789760780.895977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.895980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.895983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.895986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.895989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.895995:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.896000:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.896017:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760780.896114:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27299840<>27303936 00000100:00100000:3.0:1789760780.896246:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760780.896266:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838464:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.937977:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973984 gen 3 00000008:00010000:3.0:1789760780.938015:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760780.938021:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.938030:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.938056:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760780.938088:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838464:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760780.938200:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6665, 6665] 00000080:00200000:2.0:1789760780.938212:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.938217:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.938229:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27303936 i_blocks 55305 00000020:00200000:2.0:1789760780.938232:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.938241:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.938266:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27303936, count: 4096 00000080:00200000:2.0:1789760780.938273:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27303936, bytes: 4096 00000080:00200000:2.0:1789760780.938282:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.938288:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.938290:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27303936, 27308032) 0 0 00000080:00200000:2.0:1789760780.938293:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.938305:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6666, 6666] 00020000:00200000:2.0:1789760780.938311:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27303936 chunk: [27303936, 27308032) 27308032, 4096 00020000:00200000:2.0:1789760780.938315:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.938329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6666, 6666] 00000008:00010000:2.0:1789760780.938341:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.938346:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760780.938356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27303936, 27308032) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.938366:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27303936=0x1a0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.938446:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.938449:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27308032 00000020:00010000:2.0:1789760780.938457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.938460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.938463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.938467:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.938469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.938477:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.938482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.941756:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:3.0:1789760780.941864:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27303936<>27308032 00000100:00100000:2.0:1789760780.942471:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760780.944278:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493838592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760780.994547:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973985 gen 3 00000008:00010000:3.0:1789760780.994583:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760780.994589:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.994599:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.994675:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760780.994700:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493838592:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760780.994818:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6666, 6666] 00000080:00200000:0.0:1789760780.994832:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.994837:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.994849:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27308032 i_blocks 55305 00000020:00200000:0.0:1789760780.994852:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.994861:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.994889:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27308032, count: 4096 00000080:00200000:0.0:1789760780.994895:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27308032, bytes: 4096 00000080:00200000:0.0:1789760780.994904:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.994911:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.994913:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27308032, 27312128) 0 0 00000080:00200000:0.0:1789760780.994916:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.994929:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6667, 6667] 00020000:00200000:0.0:1789760780.994935:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27308032 chunk: [27308032, 27312128) 27312128, 4096 00020000:00200000:0.0:1789760780.994939:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.994953:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6667, 6667] 00000008:00010000:0.0:1789760780.994966:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.994970:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760780.994979:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27308032, 27312128) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.994989:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27308032=0x1a0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.995032:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.995044:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27312128 00000020:00010000:0.0:1789760780.995051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.995054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.995057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.995060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.995063:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.995069:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.995074:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.995090:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760780.995165:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 27308032<>27312128 00000100:00100000:0.0:1789760780.995205:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760780.995218:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493838720:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760781.035194:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973986 gen 3 00000008:00010000:0.0:1789760781.035230:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760781.035237:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760781.035244:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760781.035267:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760781.035294:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493838720:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.035388:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6667, 6667] 00000080:00200000:2.0:1789760781.035421:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.035427:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.035439:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27312128 i_blocks 55305 00000020:00200000:2.0:1789760781.035442:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.035452:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.035479:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27312128, count: 4096 00000080:00200000:2.0:1789760781.035485:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27312128, bytes: 4096 00000080:00200000:2.0:1789760781.035494:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.035501:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.035503:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27312128, 27316224) 0 0 00000080:00200000:2.0:1789760781.035507:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.035529:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6668, 6668] 00020000:00200000:2.0:1789760781.035536:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27312128 chunk: [27312128, 27316224) 27316224, 4096 00020000:00200000:2.0:1789760781.035539:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.035555:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6668, 6668] 00000008:00010000:2.0:1789760781.035567:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.035572:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.035580:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27312128, 27316224) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.035593:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27312128=0x1a0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.035635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.035638:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27316224 00000020:00010000:2.0:1789760781.035644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.035647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.035650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.035653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.035655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.035661:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.035667:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.035681:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760781.035753:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27312128<>27316224 00000100:00100000:2.0:1789760781.035792:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760781.035805:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.075903:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973987 gen 3 00000008:00010000:3.0:1789760781.075942:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760781.075950:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.075960:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.075977:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760781.076004:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838848:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.076174:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6668, 6668] 00000080:00200000:2.0:1789760781.076188:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.076194:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.076207:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27316224 i_blocks 55305 00000020:00200000:2.0:1789760781.076211:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.076221:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.076249:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27316224, count: 4096 00000080:00200000:2.0:1789760781.076256:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27316224, bytes: 4096 00000080:00200000:2.0:1789760781.076264:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.076271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.076273:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27316224, 27320320) 0 0 00000080:00200000:2.0:1789760781.076277:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.076290:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6669, 6669] 00020000:00200000:2.0:1789760781.076297:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27316224 chunk: [27316224, 27320320) 27320320, 4096 00020000:00200000:2.0:1789760781.076301:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.076317:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6669, 6669] 00000008:00010000:2.0:1789760781.076330:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.076335:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.076345:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27316224, 27320320) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.076356:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27316224=0x1a0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.076974:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.076979:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27320320 00000020:00010000:2.0:1789760781.076987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.076990:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.076993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.076996:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.076999:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.077006:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.077012:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.077029:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760781.077132:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27316224<>27320320 00000100:00100000:3.0:1789760781.077296:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.121629:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973988 gen 3 00000008:00010000:3.0:1789760781.121679:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760781.121686:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.121697:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.121724:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760781.121748:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493838976:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.121854:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6669, 6669] 00000080:00200000:0.0:1789760781.121868:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.121873:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.121886:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27320320 i_blocks 55305 00000020:00200000:0.0:1789760781.121890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.121900:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.121933:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27320320, count: 4096 00000080:00200000:0.0:1789760781.121940:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27320320, bytes: 4096 00000080:00200000:0.0:1789760781.121950:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.121956:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.121958:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27320320, 27324416) 0 0 00000080:00200000:0.0:1789760781.121962:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.121975:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6670, 6670] 00020000:00200000:0.0:1789760781.121981:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27320320 chunk: [27320320, 27324416) 27324416, 4096 00020000:00200000:0.0:1789760781.121986:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.122000:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6670, 6670] 00000008:00010000:0.0:1789760781.122013:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.122018:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.122027:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27320320, 27324416) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.122048:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27320320=0x1a0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.122106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.122109:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27324416 00000020:00010000:0.0:1789760781.122116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.122119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.122122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.122125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.122129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.122136:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.122141:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.122155:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760781.122232:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 27320320<>27324416 00000100:00100000:1.0:1789760781.122771:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839104:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760781.164462:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760781.164475:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.164484:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.164504:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760781.164538:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839104:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.164785:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6670, 6670] 00000080:00200000:0.0:1789760781.164799:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.164804:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.164815:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27324416 i_blocks 55305 00000020:00200000:0.0:1789760781.164819:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.164828:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.164858:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27324416, count: 4096 00000080:00200000:0.0:1789760781.164864:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27324416, bytes: 4096 00000080:00200000:0.0:1789760781.164872:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.164879:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.164881:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27324416, 27328512) 0 0 00000080:00200000:0.0:1789760781.164886:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.164899:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6671, 6671] 00020000:00200000:0.0:1789760781.164907:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27324416 chunk: [27324416, 27328512) 27328512, 4096 00020000:00200000:0.0:1789760781.164911:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.164928:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6671, 6671] 00000008:00010000:0.0:1789760781.164942:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.164946:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.164954:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27324416, 27328512) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.164965:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27324416=0x1a0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.165013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.165016:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27328512 00000020:00010000:0.0:1789760781.165023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.165027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.165029:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.165033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.165048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.165054:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.165059:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.165074:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760781.165152:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 27324416<>27328512 00000100:00100000:1.0:1789760781.165798:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493839232:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760781.243298:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973990 gen 3 00000008:00010000:1.0:1789760781.243502:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760781.243510:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.243585:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.243667:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760781.243695:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493839232:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.243773:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6671, 6671] 00000080:00200000:0.0:1789760781.243790:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.243796:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.243815:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27328512 i_blocks 55305 00000020:00200000:0.0:1789760781.243875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.243886:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.243921:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27328512, count: 4096 00000080:00200000:0.0:1789760781.243929:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27328512, bytes: 4096 00000080:00200000:0.0:1789760781.243940:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.243948:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.243950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27328512, 27332608) 0 0 00000080:00200000:0.0:1789760781.243955:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.243972:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6672, 6672] 00020000:00200000:0.0:1789760781.243980:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27328512 chunk: [27328512, 27332608) 27332608, 4096 00020000:00200000:0.0:1789760781.243986:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.244005:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6672, 6672] 00000008:00010000:0.0:1789760781.244025:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.244031:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.244053:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27328512, 27332608) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.244068:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27328512=0x1a10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.244198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.244202:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27332608 00000020:00010000:0.0:1789760781.244212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.244215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.244218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.244221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.244224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.244233:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.244240:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.244259:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760781.244795:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 27328512<>27332608 00000100:00100000:1.0:1789760781.244862:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839360:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760781.275018:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973991 gen 3 00000008:00010000:1.0:1789760781.275077:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760781.275151:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.275163:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.275225:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760781.275257:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.275344:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6672, 6672] 00000080:00200000:2.0:1789760781.275365:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.275372:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.275390:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27332608 i_blocks 55305 00000020:00200000:2.0:1789760781.275421:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.275433:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.275681:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27332608, count: 4096 00000080:00200000:2.0:1789760781.275759:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27332608, bytes: 4096 00000080:00200000:2.0:1789760781.275773:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.275782:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.275785:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27332608, 27336704) 0 0 00000080:00200000:2.0:1789760781.275790:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.275807:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6673, 6673] 00020000:00200000:2.0:1789760781.275903:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27332608 chunk: [27332608, 27336704) 27336704, 4096 00020000:00200000:2.0:1789760781.275910:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.275935:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6673, 6673] 00000008:00010000:2.0:1789760781.275954:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.275960:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.275973:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27332608, 27336704) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.275987:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27332608=0x1a11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.276064:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.276067:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27336704 00000020:00010000:2.0:1789760781.276076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.276080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.276083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.276159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.276164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.276175:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.276184:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.276205:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760781.276321:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27332608<>27336704 00000100:00100000:3.0:1789760781.276444:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493839488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.312485:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973992 gen 3 00000008:00010000:3.0:1789760781.312540:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760781.312549:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.312561:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.312573:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000080:00200000:0.0:1789760781.312721:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6673, 6673] 00000080:00200000:0.0:1789760781.312745:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.312752:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.312767:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27336704 i_blocks 55305 00000020:00200000:0.0:1789760781.312771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.312783:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.312822:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27336704, count: 4096 00000080:00200000:0.0:1789760781.312830:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27336704, bytes: 4096 00000080:00200000:0.0:1789760781.312842:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.312850:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.312853:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27336704, 27340800) 0 0 00000080:00200000:0.0:1789760781.312858:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.312874:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6674, 6674] 00020000:00200000:0.0:1789760781.312883:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27336704 chunk: [27336704, 27340800) 27340800, 4096 00020000:00200000:0.0:1789760781.312888:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.312909:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6674, 6674] 00000008:00010000:0.0:1789760781.312927:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.312933:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.312946:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27336704, 27340800) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.312962:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27336704=0x1a12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.313021:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.313024:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27340800 00000020:00010000:0.0:1789760781.313042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.313046:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.313049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.313053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.313056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.313734:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.313743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760781.313909:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493839488:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760781.313966:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:3.0:1789760781.314448:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27336704<>27340800 00000100:00100000:2.0:1789760781.314527:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760781.316506:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493839616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.368272:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973993 gen 3 00000008:00010000:3.0:1789760781.368327:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:3.0:1789760781.368336:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.368348:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.368371:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:3.0:1789760781.368498:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493839616:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.368679:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6674, 6674] 00000080:00200000:0.0:1789760781.368696:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.368703:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.368718:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27340800 i_blocks 55305 00000020:00200000:0.0:1789760781.368722:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.368740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.368775:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27340800, count: 4096 00000080:00200000:0.0:1789760781.368783:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27340800, bytes: 4096 00000080:00200000:0.0:1789760781.368793:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.368801:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.368804:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27340800, 27344896) 0 0 00000080:00200000:0.0:1789760781.368808:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.368825:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6675, 6675] 00020000:00200000:0.0:1789760781.368834:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27340800 chunk: [27340800, 27344896) 27344896, 4096 00020000:00200000:0.0:1789760781.368839:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.368858:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6675, 6675] 00000008:00010000:0.0:1789760781.368876:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.368882:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.368893:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27340800, 27344896) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.368907:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27340800=0x1a13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.368963:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.368966:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27344896 00000020:00010000:0.0:1789760781.368974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.368978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.368981:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.368984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.368986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.368993:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.369000:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.369016:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760781.369198:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 27340800<>27344896 00000100:00100000:1.0:1789760781.369274:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493839744:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760781.413948:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760781.413963:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.413975:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.414061:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760781.414263:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493839744:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.414340:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6675, 6675] 00000080:00200000:0.0:1789760781.414355:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.414362:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.414378:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27344896 i_blocks 55305 00000020:00200000:0.0:1789760781.414382:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.414413:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.414449:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27344896, count: 4096 00000080:00200000:0.0:1789760781.414457:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27344896, bytes: 4096 00000080:00200000:0.0:1789760781.414467:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.414476:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.414478:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27344896, 27348992) 0 0 00000080:00200000:0.0:1789760781.414483:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.414500:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6676, 6676] 00020000:00200000:0.0:1789760781.414509:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27344896 chunk: [27344896, 27348992) 27348992, 4096 00020000:00200000:0.0:1789760781.414514:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.414533:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6676, 6676] 00000008:00010000:0.0:1789760781.414552:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.414558:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.414570:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27344896, 27348992) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.414584:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27344896=0x1a14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.414647:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.414650:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27348992 00000020:00010000:0.0:1789760781.414658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.414661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.414664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.414668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.414670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.414678:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.414685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.414703:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760781.414873:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 27344896<>27348992 00000100:00100000:1.0:1789760781.414937:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839872:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760781.466429:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973994 gen 3 00000100:00100000:1.0:1789760781.466517:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304c380 x1876699493839872/t4294973995(4294973995) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760797 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760781.466559:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760781.466565:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.466573:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.466752:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760781.466776:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493839872:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760781.468279:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6676, 6676] 00000080:00200000:3.0:1789760781.468297:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760781.468303:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760781.468317:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27348992 i_blocks 55305 00000020:00200000:3.0:1789760781.468321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760781.468335:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760781.468362:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27348992, count: 4096 00000080:00200000:3.0:1789760781.468369:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27348992, bytes: 4096 00000080:00200000:3.0:1789760781.468379:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760781.468386:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760781.468389:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27348992, 27353088) 0 0 00000080:00200000:3.0:1789760781.468417:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760781.468432:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6677, 6677] 00020000:00200000:3.0:1789760781.468439:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27348992 chunk: [27348992, 27353088) 27353088, 4096 00020000:00200000:3.0:1789760781.468443:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760781.468459:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6677, 6677] 00000008:00010000:3.0:1789760781.468474:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760781.468479:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760781.468489:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27348992, 27353088) append 0 append lockpos 0 00000080:00200000:3.0:1789760781.468500:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27348992=0x1a15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760781.468554:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760781.468557:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27353088 00000020:00010000:3.0:1789760781.468564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760781.468567:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.468570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760781.468573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760781.468576:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760781.468583:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760781.468589:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760781.468612:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760781.468692:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27348992<>27353088 00000100:00100000:3.0:1789760781.468741:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.469350:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.503929:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973996 gen 3 00000008:00010000:3.0:1789760781.503986:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:3.0:1789760781.503993:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.504001:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.504028:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:3.0:1789760781.504062:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.504203:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6677, 6677] 00000080:00200000:2.0:1789760781.504216:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.504221:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.504233:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27353088 i_blocks 55305 00000020:00200000:2.0:1789760781.504236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.504245:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.504270:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27353088, count: 4096 00000080:00200000:2.0:1789760781.504276:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27353088, bytes: 4096 00000080:00200000:2.0:1789760781.504284:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.504292:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.504294:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27353088, 27357184) 0 0 00000080:00200000:2.0:1789760781.504298:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.504312:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6678, 6678] 00020000:00200000:2.0:1789760781.504319:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27353088 chunk: [27353088, 27357184) 27357184, 4096 00020000:00200000:2.0:1789760781.504323:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.504338:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6678, 6678] 00000008:00010000:2.0:1789760781.504351:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.504356:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.504366:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27353088, 27357184) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.504377:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27353088=0x1a16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.504460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.504463:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27357184 00000020:00010000:2.0:1789760781.504470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.504474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.504477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.504480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.504483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.504489:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.504495:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.504507:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760781.504586:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27353088<>27357184 00000100:00100000:2.0:1789760781.504622:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760781.504636:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.541613:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973997 gen 3 00000008:00010000:3.0:1789760781.541671:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760781.541680:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.541692:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.541712:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760781.541747:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840128:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.541750:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6678, 6678] 00000080:00200000:2.0:1789760781.541773:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.541780:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.541846:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27357184 i_blocks 55305 00000020:00200000:2.0:1789760781.541851:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.541864:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.541902:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27357184, count: 4096 00000080:00200000:2.0:1789760781.541909:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27357184, bytes: 4096 00000080:00200000:2.0:1789760781.541972:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.541981:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.541984:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27357184, 27361280) 0 0 00000080:00200000:2.0:1789760781.541989:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.542005:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6679, 6679] 00020000:00200000:2.0:1789760781.542014:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27357184 chunk: [27357184, 27361280) 27361280, 4096 00020000:00200000:2.0:1789760781.542018:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.542046:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6679, 6679] 00000008:00010000:2.0:1789760781.542063:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.542069:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.542080:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27357184, 27361280) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.542153:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27357184=0x1a17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.542213:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.542217:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27361280 00000020:00010000:2.0:1789760781.542225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.542228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.542231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.542235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.542238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.542253:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.542259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.542278:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760781.542375:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 27357184<>27361280 00000100:00100000:3.0:1789760781.542441:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840256:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760781.549065:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760781.570548:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973998 gen 3 00000008:00010000:3.0:1789760781.570600:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760781.570608:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.570626:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.570702:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760781.570732:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.570820:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6679, 6679] 00000080:00200000:2.0:1789760781.570837:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.570844:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.570860:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27361280 i_blocks 55305 00000020:00200000:2.0:1789760781.570864:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.570875:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.570908:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27361280, count: 4096 00000080:00200000:2.0:1789760781.570915:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27361280, bytes: 4096 00000080:00200000:2.0:1789760781.570927:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.570937:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.570939:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27361280, 27365376) 0 0 00000080:00200000:2.0:1789760781.570944:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.570960:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6680, 6680] 00020000:00200000:2.0:1789760781.570969:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27361280 chunk: [27361280, 27365376) 27365376, 4096 00020000:00200000:2.0:1789760781.570974:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.570993:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6680, 6680] 00000008:00010000:2.0:1789760781.571010:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.571017:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.571028:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27361280, 27365376) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.571052:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27361280=0x1a18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.571111:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.571114:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27365376 00000020:00010000:2.0:1789760781.571122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.571126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.571129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.571133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.571136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.571144:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.571150:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.571165:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760781.571322:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 27361280<>27365376 00000100:00100000:3.0:1789760781.571411:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.601109:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294973999 gen 3 00000008:00010000:3.0:1789760781.601161:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760781.601171:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.601184:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.601210:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760781.601241:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.601424:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6680, 6680] 00000080:00200000:2.0:1789760781.601440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.601446:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.601462:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27365376 i_blocks 55305 00000020:00200000:2.0:1789760781.601465:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.601478:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.601595:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27365376, count: 4096 00000080:00200000:2.0:1789760781.601603:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27365376, bytes: 4096 00000080:00200000:2.0:1789760781.601620:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.601630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.601632:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27365376, 27369472) 0 0 00000080:00200000:2.0:1789760781.601638:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.601656:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6681, 6681] 00020000:00200000:2.0:1789760781.601667:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27365376 chunk: [27365376, 27369472) 27369472, 4096 00020000:00200000:2.0:1789760781.601672:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.601765:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6681, 6681] 00000008:00010000:2.0:1789760781.601786:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.601793:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.601807:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27365376, 27369472) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.601822:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27365376=0x1a19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.602663:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.602670:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27369472 00000020:00010000:2.0:1789760781.602683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.602770:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.602774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.602779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.602782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.602793:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.602802:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.602828:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760781.602981:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 27365376<>27369472 00000100:00100000:3.0:1789760781.603201:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760781.603230:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.656958:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974000 gen 3 00000008:00010000:3.0:1789760781.657150:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760781.657159:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.657171:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.657235:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760781.657266:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.657354:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6681, 6681] 00000080:00200000:2.0:1789760781.657371:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.657377:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.657417:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27369472 i_blocks 55305 00000020:00200000:2.0:1789760781.657421:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.657432:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.657465:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27369472, count: 4096 00000080:00200000:2.0:1789760781.657472:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27369472, bytes: 4096 00000080:00200000:2.0:1789760781.657483:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.657493:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.657495:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27369472, 27373568) 0 0 00000080:00200000:2.0:1789760781.657500:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.657632:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6682, 6682] 00020000:00200000:2.0:1789760781.657642:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27369472 chunk: [27369472, 27373568) 27373568, 4096 00020000:00200000:2.0:1789760781.657647:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.657667:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6682, 6682] 00000008:00010000:2.0:1789760781.657753:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.657760:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.657774:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27369472, 27373568) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.657789:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27369472=0x1a1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.657849:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.657853:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27373568 00000020:00010000:2.0:1789760781.657862:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.657865:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.657868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.657872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.657875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.657882:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.657891:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.657907:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760781.658149:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27369472<>27373568 00000100:00100000:3.0:1789760781.658225:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.707293:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974001 gen 3 00000008:00010000:3.0:1789760781.707329:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760781.707336:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.707345:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.707365:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760781.707390:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.707622:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6682, 6682] 00000080:00200000:2.0:1789760781.707636:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.707641:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.707654:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27373568 i_blocks 55305 00000020:00200000:2.0:1789760781.707657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.707666:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.707693:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27373568, count: 4096 00000080:00200000:2.0:1789760781.707699:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27373568, bytes: 4096 00000080:00200000:2.0:1789760781.707707:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.707725:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.707727:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27373568, 27377664) 0 0 00000080:00200000:2.0:1789760781.707731:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.707744:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6683, 6683] 00020000:00200000:2.0:1789760781.707751:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27373568 chunk: [27373568, 27377664) 27377664, 4096 00020000:00200000:2.0:1789760781.707755:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.707771:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6683, 6683] 00000008:00010000:2.0:1789760781.707783:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.707788:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.707796:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27373568, 27377664) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.707807:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27373568=0x1a1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.707851:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.707854:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27377664 00000020:00010000:2.0:1789760781.707860:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.707863:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.707866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.707869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.707872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.707878:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.707884:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.707897:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760781.707973:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27373568<>27377664 00000100:00100000:3.0:1789760781.708048:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840768:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.750936:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974002 gen 3 00000008:00010000:3.0:1789760781.750980:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760781.750987:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.750996:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.751016:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760781.751053:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493840768:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760781.751158:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6683, 6683] 00000080:00200000:2.0:1789760781.751170:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.751175:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.751186:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27377664 i_blocks 55305 00000020:00200000:2.0:1789760781.751189:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.751198:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.751227:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27377664, count: 4096 00000080:00200000:2.0:1789760781.751233:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27377664, bytes: 4096 00000080:00200000:2.0:1789760781.751241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.751247:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.751250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27377664, 27381760) 0 0 00000080:00200000:2.0:1789760781.751254:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.751266:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6684, 6684] 00020000:00200000:2.0:1789760781.751273:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27377664 chunk: [27377664, 27381760) 27381760, 4096 00020000:00200000:2.0:1789760781.751277:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.751292:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6684, 6684] 00000008:00010000:2.0:1789760781.751305:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.751310:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760781.751319:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27377664, 27381760) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.751329:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27377664=0x1a1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.751372:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.751375:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27381760 00000020:00010000:2.0:1789760781.751382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.751385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.751388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.751391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.751433:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.751440:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.751446:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.751460:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760781.751551:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27377664<>27381760 00000100:00100000:2.0:1789760781.751597:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840896:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760781.805265:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974003 gen 3 00000008:00010000:3.0:1789760781.805310:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760781.805317:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.805328:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.805352:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760781.805379:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493840896:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760781.805428:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6684, 6684] 00000080:00200000:1.0:1789760781.805445:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.805451:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.805464:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27381760 i_blocks 55305 00000020:00200000:1.0:1789760781.805468:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.805479:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.805505:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27381760, count: 4096 00000080:00200000:1.0:1789760781.805512:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27381760, bytes: 4096 00000080:00200000:1.0:1789760781.805519:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.805527:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.805529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27381760, 27385856) 0 0 00000080:00200000:1.0:1789760781.805534:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.805546:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6685, 6685] 00020000:00200000:1.0:1789760781.805553:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27381760 chunk: [27381760, 27385856) 27385856, 4096 00020000:00200000:1.0:1789760781.805557:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.805571:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6685, 6685] 00000008:00010000:1.0:1789760781.805583:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.805588:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760781.805597:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27381760, 27385856) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.805814:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27381760=0x1a1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.805937:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.805940:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27385856 00000020:00010000:1.0:1789760781.805948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.805952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.805955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.805959:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.805961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.805971:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.805977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.805993:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760781.806086:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 27381760<>27385856 00000100:00100000:0.0:1789760781.806156:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841024:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760781.848985:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974004 gen 3 00000008:00010000:0.0:1789760781.849024:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760781.849040:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760781.849048:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760781.849070:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760781.849105:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841024:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760781.849247:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6685, 6685] 00000080:00200000:1.0:1789760781.849260:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.849265:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.849277:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27385856 i_blocks 55305 00000020:00200000:1.0:1789760781.849280:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.849288:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.849315:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27385856, count: 4096 00000080:00200000:1.0:1789760781.849320:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27385856, bytes: 4096 00000080:00200000:1.0:1789760781.849328:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.849335:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.849337:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27385856, 27389952) 0 0 00000080:00200000:1.0:1789760781.849345:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.849358:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6686, 6686] 00020000:00200000:1.0:1789760781.849365:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27385856 chunk: [27385856, 27389952) 27389952, 4096 00020000:00200000:1.0:1789760781.849369:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.849383:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6686, 6686] 00000008:00010000:1.0:1789760781.849467:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.849473:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760781.849483:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27385856, 27389952) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.849492:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27385856=0x1a1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.849667:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.849670:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27389952 00000020:00010000:1.0:1789760781.849677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.849680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.849684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.849687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.849689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.849697:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.849703:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.849718:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e923c0 00000008:00100000:1.0:1789760781.849803:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 27385856<>27389952 00000100:00100000:0.0:1789760781.849873:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493841152:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760781.894950:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974005 gen 3 00000008:00010000:0.0:1789760781.894982:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e923c0 00000020:00200000:0.0:1789760781.894988:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760781.894996:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760781.895015:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e923c0 00000100:00100000:0.0:1789760781.895046:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493841152:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760781.895080:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6686, 6686] 00000080:00200000:1.0:1789760781.895104:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760781.895110:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760781.895122:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27389952 i_blocks 55305 00000020:00200000:1.0:1789760781.895126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760781.895136:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760781.895163:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27389952, count: 4096 00000080:00200000:1.0:1789760781.895169:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27389952, bytes: 4096 00000080:00200000:1.0:1789760781.895182:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760781.895190:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760781.895191:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27389952, 27394048) 0 0 00000080:00200000:1.0:1789760781.895195:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760781.895208:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6687, 6687] 00020000:00200000:1.0:1789760781.895215:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27389952 chunk: [27389952, 27394048) 27394048, 4096 00020000:00200000:1.0:1789760781.895220:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760781.895236:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6687, 6687] 00000008:00010000:1.0:1789760781.895250:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760781.895254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760781.895264:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27389952, 27394048) append 0 append lockpos 0 00000080:00200000:1.0:1789760781.895274:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27389952=0x1a1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760781.895329:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760781.895332:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27394048 00000020:00010000:1.0:1789760781.895339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760781.895342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.895345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760781.895348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760781.895351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760781.895359:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760781.895365:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760781.895380:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760781.895591:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27389952<>27394048 00000100:00100000:1.0:1789760781.895639:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841280:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760781.951799:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974006 gen 3 00000008:00010000:1.0:1789760781.951838:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760781.951845:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.951853:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.951873:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760781.951899:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841280:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760781.952006:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6687, 6687] 00000080:00200000:0.0:1789760781.952019:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.952024:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.952047:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27394048 i_blocks 55305 00000020:00200000:0.0:1789760781.952050:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.952059:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.952086:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27394048, count: 4096 00000080:00200000:0.0:1789760781.952092:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27394048, bytes: 4096 00000080:00200000:0.0:1789760781.952102:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.952109:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.952111:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27394048, 27398144) 0 0 00000080:00200000:0.0:1789760781.952115:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.952128:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6688, 6688] 00020000:00200000:0.0:1789760781.952134:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27394048 chunk: [27394048, 27398144) 27398144, 4096 00020000:00200000:0.0:1789760781.952138:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.952153:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6688, 6688] 00000008:00010000:0.0:1789760781.952166:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.952170:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760781.952179:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27394048, 27398144) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.952190:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27394048=0x1a20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.952236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.952239:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27398144 00000020:00010000:0.0:1789760781.952246:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.952249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.952252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.952256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.952258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.952265:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.952271:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.952286:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760781.952361:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 27394048<>27398144 00000100:00100000:1.0:1789760781.952667:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760781.952681:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841408:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760782.006934:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760782.006945:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.006954:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.007020:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760782.007057:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841408:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.009801:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6688, 6688] 00000080:00200000:0.0:1789760782.009824:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.009830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.009844:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27398144 i_blocks 55305 00000020:00200000:0.0:1789760782.009848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.009862:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.009889:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27398144, count: 4096 00000080:00200000:0.0:1789760782.009896:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27398144, bytes: 4096 00000080:00200000:0.0:1789760782.009908:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.009916:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.009918:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27398144, 27402240) 0 0 00000080:00200000:0.0:1789760782.009922:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.009935:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6689, 6689] 00020000:00200000:0.0:1789760782.009943:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27398144 chunk: [27398144, 27402240) 27402240, 4096 00020000:00200000:0.0:1789760782.009946:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.009962:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6689, 6689] 00000008:00010000:0.0:1789760782.009977:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.009982:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.009991:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27398144, 27402240) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.010003:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27398144=0x1a21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.010075:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.010077:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27402240 00000020:00010000:0.0:1789760782.010084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.010155:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.010158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.010162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.010164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.010174:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.010180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.010206:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760782.010298:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 27398144<>27402240 00000100:00100000:1.0:1789760782.010851:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841536:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.047919:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974008 gen 3 00000008:00010000:1.0:1789760782.047966:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760782.047973:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.047980:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.048002:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760782.048042:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841536:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.048267:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6689, 6689] 00000080:00200000:0.0:1789760782.048282:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.048287:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.048298:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27402240 i_blocks 55305 00000020:00200000:0.0:1789760782.048301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.048310:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.048339:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27402240, count: 4096 00000080:00200000:0.0:1789760782.048344:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27402240, bytes: 4096 00000080:00200000:0.0:1789760782.048353:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.048359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.048499:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27402240, 27406336) 0 0 00000080:00200000:0.0:1789760782.048705:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.048718:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6690, 6690] 00020000:00200000:0.0:1789760782.048726:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27402240 chunk: [27402240, 27406336) 27406336, 4096 00020000:00200000:0.0:1789760782.048729:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.048744:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6690, 6690] 00000008:00010000:0.0:1789760782.048757:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.048762:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.048772:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27402240, 27406336) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.048783:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27402240=0x1a22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.048827:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.048830:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27406336 00000020:00010000:0.0:1789760782.048837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.048840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.048843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.048846:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.048848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.048855:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.048861:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.048878:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760782.048955:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 27402240<>27406336 00000100:00100000:1.0:1789760782.049543:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.049562:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841664:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.119259:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974009 gen 3 00000008:00010000:1.0:1789760782.119296:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760782.119303:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.119312:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.119334:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760782.119361:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841664:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.119721:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6690, 6690] 00000080:00200000:0.0:1789760782.119735:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.119739:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.119751:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27406336 i_blocks 55305 00000020:00200000:0.0:1789760782.119754:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.119763:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.119794:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27406336, count: 4096 00000080:00200000:0.0:1789760782.119800:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27406336, bytes: 4096 00000080:00200000:0.0:1789760782.119809:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.119816:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.119818:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27406336, 27410432) 0 0 00000080:00200000:0.0:1789760782.119822:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.119835:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6691, 6691] 00020000:00200000:0.0:1789760782.119842:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27406336 chunk: [27406336, 27410432) 27410432, 4096 00020000:00200000:0.0:1789760782.119846:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.119861:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6691, 6691] 00000008:00010000:0.0:1789760782.119875:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.119880:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.119889:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27406336, 27410432) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.119900:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27406336=0x1a23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.119945:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.119947:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27410432 00000020:00010000:0.0:1789760782.119954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.119957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.119960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.119963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.119965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.119972:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.119977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.119991:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760782.120082:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 27406336<>27410432 00000100:00100000:1.0:1789760782.120301:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841792:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.176828:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974010 gen 3 00000008:00010000:1.0:1789760782.176864:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760782.176870:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.176878:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.176898:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760782.176920:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841792:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.176938:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6691, 6691] 00000080:00200000:0.0:1789760782.176961:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.176968:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.176980:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27410432 i_blocks 55305 00000020:00200000:0.0:1789760782.176984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.176992:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.177020:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27410432, count: 4096 00000080:00200000:0.0:1789760782.177035:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27410432, bytes: 4096 00000080:00200000:0.0:1789760782.177044:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.177051:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.177053:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27410432, 27414528) 0 0 00000080:00200000:0.0:1789760782.177057:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.177070:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6692, 6692] 00020000:00200000:0.0:1789760782.177078:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27410432 chunk: [27410432, 27414528) 27414528, 4096 00020000:00200000:0.0:1789760782.177082:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.177096:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6692, 6692] 00000008:00010000:0.0:1789760782.177110:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.177114:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.177124:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27410432, 27414528) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.177134:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27410432=0x1a24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.177176:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.177179:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27414528 00000020:00010000:0.0:1789760782.177185:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.177189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.177191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.177194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.177197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.177204:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.177209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.177224:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760782.177299:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 27410432<>27414528 00000100:00100000:1.0:1789760782.178227:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.178246:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841920:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760782.241229:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760782.241241:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.241250:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.241280:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760782.241308:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493841920:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.241697:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6692, 6692] 00000080:00200000:0.0:1789760782.241709:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.241725:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.241736:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27414528 i_blocks 55305 00000020:00200000:0.0:1789760782.241740:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.241749:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.241776:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27414528, count: 4096 00000080:00200000:0.0:1789760782.241782:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27414528, bytes: 4096 00000080:00200000:0.0:1789760782.241790:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.241797:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.241799:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27414528, 27418624) 0 0 00000080:00200000:0.0:1789760782.241803:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.241816:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6693, 6693] 00020000:00200000:0.0:1789760782.241824:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27414528 chunk: [27414528, 27418624) 27418624, 4096 00020000:00200000:0.0:1789760782.241829:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.241845:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6693, 6693] 00000008:00010000:0.0:1789760782.241858:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.241863:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.241872:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27414528, 27418624) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.241882:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27414528=0x1a25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.241933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.241935:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27418624 00000020:00010000:0.0:1789760782.241941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.241944:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.241947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.241950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.241953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.241960:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.241965:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.241978:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760782.242067:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 27414528<>27418624 00000100:00100000:1.0:1789760782.242496:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842048:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.305663:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974012 gen 3 00000008:00010000:1.0:1789760782.305711:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760782.305718:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.305727:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.305748:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760782.305774:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842048:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.305937:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6693, 6693] 00000080:00200000:0.0:1789760782.305950:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.305955:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.305967:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27418624 i_blocks 55305 00000020:00200000:0.0:1789760782.305971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.305980:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.306007:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27418624, count: 4096 00000080:00200000:0.0:1789760782.306013:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27418624, bytes: 4096 00000080:00200000:0.0:1789760782.306021:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.306037:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.306039:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27418624, 27422720) 0 0 00000080:00200000:0.0:1789760782.306043:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.306055:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6694, 6694] 00020000:00200000:0.0:1789760782.306062:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27418624 chunk: [27418624, 27422720) 27422720, 4096 00020000:00200000:0.0:1789760782.306066:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.306081:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6694, 6694] 00000008:00010000:0.0:1789760782.306095:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.306100:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.306111:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27418624, 27422720) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.306121:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27418624=0x1a26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.306167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.306169:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27422720 00000020:00010000:0.0:1789760782.306176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.306180:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.306183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.306186:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.306189:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.306196:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.306201:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.306275:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760782.306364:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 27418624<>27422720 00000100:00100000:1.0:1789760782.306432:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842176:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.390212:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974013 gen 3 00000008:00010000:1.0:1789760782.390252:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760782.390260:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.390268:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.390290:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760782.390318:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842176:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.390325:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6694, 6694] 00000080:00200000:0.0:1789760782.390341:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.390346:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.390358:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27422720 i_blocks 55305 00000020:00200000:0.0:1789760782.390362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.390371:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.390433:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27422720, count: 4096 00000080:00200000:0.0:1789760782.390440:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27422720, bytes: 4096 00000080:00200000:0.0:1789760782.390447:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.390455:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.390457:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27422720, 27426816) 0 0 00000080:00200000:0.0:1789760782.390461:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.390473:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6695, 6695] 00020000:00200000:0.0:1789760782.390481:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27422720 chunk: [27422720, 27426816) 27426816, 4096 00020000:00200000:0.0:1789760782.390485:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.390501:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6695, 6695] 00000008:00010000:0.0:1789760782.390514:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.390519:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.390529:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27422720, 27426816) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.390552:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27422720=0x1a27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.390595:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.390598:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27426816 00000020:00010000:0.0:1789760782.390604:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.390607:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.390609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.390611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.390613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.390618:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.390622:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.390635:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760782.390720:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 27422720<>27426816 00000100:00100000:1.0:1789760782.390789:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842304:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.434871:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974014 gen 3 00000008:00010000:1.0:1789760782.434909:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760782.434915:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.434923:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.435004:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760782.435047:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842304:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.435283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6695, 6695] 00000080:00200000:0.0:1789760782.435295:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.435300:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.435311:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27426816 i_blocks 55305 00000020:00200000:0.0:1789760782.435314:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.435332:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.435359:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27426816, count: 4096 00000080:00200000:0.0:1789760782.435365:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27426816, bytes: 4096 00000080:00200000:0.0:1789760782.435382:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.435390:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.435425:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27426816, 27430912) 0 0 00000080:00200000:0.0:1789760782.435430:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.435442:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6696, 6696] 00020000:00200000:0.0:1789760782.435449:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27426816 chunk: [27426816, 27430912) 27430912, 4096 00020000:00200000:0.0:1789760782.435453:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.436021:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6696, 6696] 00000008:00010000:0.0:1789760782.436200:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.436206:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.436216:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27426816, 27430912) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.436228:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27426816=0x1a28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.436283:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.436286:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27430912 00000020:00010000:0.0:1789760782.436292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.436296:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.436299:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.436302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.436305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.439037:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.439055:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.439093:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760782.439192:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 27426816<>27430912 00000100:00100000:1.0:1789760782.439275:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842432:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760782.486315:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760782.486325:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.486334:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.486353:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760782.486379:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842432:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.486551:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6696, 6696] 00000080:00200000:0.0:1789760782.486564:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.486569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.486580:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27430912 i_blocks 55305 00000020:00200000:0.0:1789760782.486583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.486591:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.486619:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27430912, count: 4096 00000080:00200000:0.0:1789760782.486630:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27430912, bytes: 4096 00000080:00200000:0.0:1789760782.486639:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.486645:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.486647:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27430912, 27435008) 0 0 00000080:00200000:0.0:1789760782.486651:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.486663:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6697, 6697] 00020000:00200000:0.0:1789760782.486669:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27430912 chunk: [27430912, 27435008) 27435008, 4096 00020000:00200000:0.0:1789760782.486673:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.486686:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6697, 6697] 00000008:00010000:0.0:1789760782.486699:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.486704:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.486711:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27430912, 27435008) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.486721:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27430912=0x1a29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.486767:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.486769:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27435008 00000020:00010000:0.0:1789760782.486775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.486778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.486781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.486784:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.486787:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.486792:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.486796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760782.486963:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:1.0:1789760782.487049:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27430912<>27435008 00000100:00100000:1.0:1789760782.487078:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.487089:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842560:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.524770:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974016 gen 3 00000008:00010000:1.0:1789760782.524819:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760782.524827:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.524836:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.524859:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760782.524886:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842560:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.525149:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6697, 6697] 00000080:00200000:0.0:1789760782.525162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.525168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.525179:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27435008 i_blocks 55305 00000020:00200000:0.0:1789760782.525181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.525189:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.525216:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27435008, count: 4096 00000080:00200000:0.0:1789760782.525222:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27435008, bytes: 4096 00000080:00200000:0.0:1789760782.525229:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.525236:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.525237:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27435008, 27439104) 0 0 00000080:00200000:0.0:1789760782.525241:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.525255:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6698, 6698] 00020000:00200000:0.0:1789760782.525263:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27435008 chunk: [27435008, 27439104) 27439104, 4096 00020000:00200000:0.0:1789760782.525266:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.525282:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6698, 6698] 00000008:00010000:0.0:1789760782.525295:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.525300:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.525310:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27435008, 27439104) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.525320:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27435008=0x1a2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.525365:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.525368:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27439104 00000020:00010000:0.0:1789760782.525375:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.525378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.525381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.525384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.525386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.525475:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.525482:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.525499:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760782.525641:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 27435008<>27439104 00000100:00100000:1.0:1789760782.525697:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842688:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760782.572902:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760782.578203:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974017 gen 3 00000008:00010000:1.0:1789760782.578237:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760782.578244:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.578252:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.578268:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760782.578294:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493842688:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.578313:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6698, 6698] 00000080:00200000:0.0:1789760782.578327:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.578332:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.578344:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27439104 i_blocks 55305 00000020:00200000:0.0:1789760782.578347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.578356:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.578384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27439104, count: 4096 00000080:00200000:0.0:1789760782.578390:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27439104, bytes: 4096 00000080:00200000:0.0:1789760782.578426:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.578434:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.578436:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27439104, 27443200) 0 0 00000080:00200000:0.0:1789760782.578440:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.578454:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6699, 6699] 00020000:00200000:0.0:1789760782.578461:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27439104 chunk: [27439104, 27443200) 27443200, 4096 00020000:00200000:0.0:1789760782.578464:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.578479:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6699, 6699] 00000008:00010000:0.0:1789760782.578492:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.578497:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.578506:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27439104, 27443200) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.578516:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27439104=0x1a2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.578560:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.578563:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27443200 00000020:00010000:0.0:1789760782.578570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.578573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.578575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.578578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.578581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.578587:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.578592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.578607:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760782.578684:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 27439104<>27443200 00000100:00100000:1.0:1789760782.578856:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.641171:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974018 gen 3 00000008:00010000:1.0:1789760782.641207:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760782.641214:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.641222:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.641301:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000080:00200000:0.0:1789760782.641306:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6699, 6699] 00000080:00200000:0.0:1789760782.641319:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.641324:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760782.641326:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842816:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.641335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27443200 i_blocks 55305 00000020:00200000:0.0:1789760782.641338:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.641347:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.641374:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27443200, count: 4096 00000080:00200000:0.0:1789760782.641380:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27443200, bytes: 4096 00000080:00200000:0.0:1789760782.641389:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.641415:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.641416:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27443200, 27447296) 0 0 00000080:00200000:0.0:1789760782.641419:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.641430:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6700, 6700] 00020000:00200000:0.0:1789760782.641435:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27443200 chunk: [27443200, 27447296) 27447296, 4096 00020000:00200000:0.0:1789760782.641437:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.641449:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6700, 6700] 00000008:00010000:0.0:1789760782.641459:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.641463:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.641470:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27443200, 27447296) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.641478:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27443200=0x1a2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.641637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.641639:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27447296 00000020:00010000:0.0:1789760782.641646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.641650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.641653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.641656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.641658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.641665:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.641671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.641688:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760782.641764:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 27443200<>27447296 00000100:00100000:1.0:1789760782.642115:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.642129:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842944:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.680720:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974019 gen 3 00000008:00010000:1.0:1789760782.680754:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760782.680760:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.680767:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.680846:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760782.680875:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493842944:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.680955:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6700, 6700] 00000080:00200000:0.0:1789760782.680968:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.680973:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.680984:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27447296 i_blocks 55305 00000020:00200000:0.0:1789760782.680987:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.680996:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.681079:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27447296, count: 4096 00000080:00200000:0.0:1789760782.681086:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27447296, bytes: 4096 00000080:00200000:0.0:1789760782.681094:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.681101:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.681103:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27447296, 27451392) 0 0 00000080:00200000:0.0:1789760782.681106:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.681118:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6701, 6701] 00020000:00200000:0.0:1789760782.681124:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27447296 chunk: [27447296, 27451392) 27451392, 4096 00020000:00200000:0.0:1789760782.681128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.681143:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6701, 6701] 00000008:00010000:0.0:1789760782.681157:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.681162:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.681172:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27447296, 27451392) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.681184:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27447296=0x1a2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.681229:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.681231:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27451392 00000020:00010000:0.0:1789760782.681238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.681241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.681244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.681247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.681249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.681254:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.681260:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.681275:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760782.681349:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 27447296<>27451392 00000100:00100000:1.0:1789760782.681435:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493843072:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.738931:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974020 gen 3 00000008:00010000:1.0:1789760782.738967:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760782.738973:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.738981:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.739002:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760782.739037:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493843072:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.739147:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6701, 6701] 00000080:00200000:0.0:1789760782.739161:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.739165:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.739178:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27451392 i_blocks 55305 00000020:00200000:0.0:1789760782.739181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.739189:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.739217:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27451392, count: 4096 00000080:00200000:0.0:1789760782.739222:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27451392, bytes: 4096 00000080:00200000:0.0:1789760782.739230:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.739237:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.739239:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27451392, 27455488) 0 0 00000080:00200000:0.0:1789760782.739242:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.739255:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6702, 6702] 00020000:00200000:0.0:1789760782.739261:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27451392 chunk: [27451392, 27455488) 27455488, 4096 00020000:00200000:0.0:1789760782.739265:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.739279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6702, 6702] 00000008:00010000:0.0:1789760782.739292:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.739296:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.739306:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27451392, 27455488) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.739317:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27451392=0x1a2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.739362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.739364:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27455488 00000020:00010000:0.0:1789760782.739370:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.739374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.739376:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.739379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.739382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.739389:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.739487:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.739505:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760782.739582:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 27451392<>27455488 00000100:00100000:1.0:1789760782.739710:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843200:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.821833:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974021 gen 3 00000008:00010000:1.0:1789760782.821870:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760782.821876:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.821884:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.821906:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760782.821932:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843200:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760782.822145:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6702, 6702] 00000080:00200000:0.0:1789760782.822222:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.822228:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.822239:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27455488 i_blocks 55305 00000020:00200000:0.0:1789760782.822243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.822251:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.822278:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27455488, count: 4096 00000080:00200000:0.0:1789760782.822283:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27455488, bytes: 4096 00000080:00200000:0.0:1789760782.822291:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.822299:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.822301:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27455488, 27459584) 0 0 00000080:00200000:0.0:1789760782.822305:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.822318:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6703, 6703] 00020000:00200000:0.0:1789760782.822325:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27455488 chunk: [27455488, 27459584) 27459584, 4096 00020000:00200000:0.0:1789760782.822329:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.822617:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6703, 6703] 00000008:00010000:0.0:1789760782.822632:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.822638:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760782.822647:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27455488, 27459584) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.822657:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27455488=0x1a2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.822902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.822905:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27459584 00000020:00010000:0.0:1789760782.822913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.822917:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.822919:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.822923:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.822925:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.822931:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.823117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.823137:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760782.831132:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 27455488<>27459584 00000100:00100000:1.0:1789760782.831505:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.831696:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843328:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760782.893971:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493843456:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760782.895427:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.895496:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843584:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760782.896900:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.896917:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2368:1876699493843712:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760782.898893:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493843456:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760782.902353:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843584:192.168.203.160@tcp:400:kworker.0 00000100:00100000:0.0:1789760782.902518:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2368:1876699493843712:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760782.917347:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974022 gen 3 00000008:00010000:1.0:1789760782.917385:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760782.917391:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.917430:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.917451:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760782.917477:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493843328:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760782.917639:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6703, 6703] 00000080:00200000:3.0:1789760782.917654:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.917660:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.917673:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27459584 i_blocks 55305 00000020:00200000:3.0:1789760782.917676:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.917686:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.917715:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27459584, count: 4096 00000080:00200000:3.0:1789760782.917721:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27459584, bytes: 4096 00000080:00200000:3.0:1789760782.917730:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.917738:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.917740:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27459584, 27463680) 0 0 00000080:00200000:3.0:1789760782.917743:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.917757:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6704, 6704] 00020000:00200000:3.0:1789760782.917764:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27459584 chunk: [27459584, 27463680) 27463680, 4096 00020000:00200000:3.0:1789760782.917768:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.917785:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6704, 6704] 00000008:00010000:3.0:1789760782.917800:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.917806:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760782.917899:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27459584, 27463680) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.917912:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27459584=0x1a30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.917960:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.917963:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27463680 00000020:00010000:3.0:1789760782.917970:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.917974:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.917976:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.917979:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.917982:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.917988:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.917993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.918008:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760782.918172:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 27459584<>27463680 00000100:00100000:2.0:1789760782.918248:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493843840:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760782.994989:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760782.995000:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.995009:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.995041:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760782.995067:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493843840:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760782.995272:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6704, 6704] 00000080:00200000:1.0:1789760782.995286:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760782.995291:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760782.995302:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27463680 i_blocks 55305 00000020:00200000:1.0:1789760782.995306:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760782.995314:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760782.995341:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27463680, count: 4096 00000080:00200000:1.0:1789760782.995348:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27463680, bytes: 4096 00000080:00200000:1.0:1789760782.995357:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760782.995364:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760782.995366:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27463680, 27467776) 0 0 00000080:00200000:1.0:1789760782.995370:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760782.995382:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6705, 6705] 00020000:00200000:1.0:1789760782.995389:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27463680 chunk: [27463680, 27467776) 27467776, 4096 00020000:00200000:1.0:1789760782.995426:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760782.995443:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6705, 6705] 00000008:00010000:1.0:1789760782.995456:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760782.995462:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760782.995470:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27463680, 27467776) append 0 append lockpos 0 00000080:00200000:1.0:1789760782.995481:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27463680=0x1a31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760782.995525:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760782.995528:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27467776 00000020:00010000:1.0:1789760782.995610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760782.995615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.995618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.995621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760782.995623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760782.995632:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760782.995637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760782.995652:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760782.995729:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27463680<>27467776 00000100:00100000:0.0:1789760782.995808:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493843968:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760783.036071:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974024 gen 3 00000008:00010000:1.0:1789760783.036181:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760783.036187:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.036195:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.036221:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760783.036245:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493843968:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760783.036360:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6705, 6705] 00000080:00200000:2.0:1789760783.036372:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.036376:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.036387:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27467776 i_blocks 55305 00000020:00200000:2.0:1789760783.036390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.036420:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.036447:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27467776, count: 4096 00000080:00200000:2.0:1789760783.036453:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27467776, bytes: 4096 00000080:00200000:2.0:1789760783.036463:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.036469:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.036471:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27467776, 27471872) 0 0 00000080:00200000:2.0:1789760783.036475:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.036488:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6706, 6706] 00020000:00200000:2.0:1789760783.036495:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27467776 chunk: [27467776, 27471872) 27471872, 4096 00020000:00200000:2.0:1789760783.036498:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.036579:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6706, 6706] 00000008:00010000:2.0:1789760783.036593:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.036598:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760783.036606:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27467776, 27471872) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.036618:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27467776=0x1a32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.036660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.036663:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27471872 00000020:00010000:2.0:1789760783.036670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.036673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.036676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.036679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.036683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.036690:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.036696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.036709:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760783.037184:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27467776<>27471872 00000100:00100000:3.0:1789760783.037258:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844096:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.090892:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974025 gen 3 00000008:00010000:2.0:1789760783.090926:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:2.0:1789760783.090933:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.090942:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.090964:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:2.0:1789760783.090988:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844096:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760783.094213:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6706, 6706] 00000080:00200000:1.0:1789760783.094235:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760783.094241:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760783.094255:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27471872 i_blocks 55305 00000020:00200000:1.0:1789760783.094260:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760783.094274:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760783.094302:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27471872, count: 4096 00000080:00200000:1.0:1789760783.094309:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27471872, bytes: 4096 00000080:00200000:1.0:1789760783.094320:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760783.094327:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760783.094330:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27471872, 27475968) 0 0 00000080:00200000:1.0:1789760783.094333:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760783.094347:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6707, 6707] 00020000:00200000:1.0:1789760783.094354:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27471872 chunk: [27471872, 27475968) 27475968, 4096 00020000:00200000:1.0:1789760783.094357:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.094374:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6707, 6707] 00000008:00010000:1.0:1789760783.094388:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760783.094430:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760783.094442:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27471872, 27475968) append 0 append lockpos 0 00000080:00200000:1.0:1789760783.094454:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27471872=0x1a33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760783.094692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760783.094704:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27475968 00000020:00010000:1.0:1789760783.094713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760783.094716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.094719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.094721:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760783.094724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760783.094734:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760783.094740:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760783.094760:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760783.095008:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 27471872<>27475968 00000100:00100000:0.0:1789760783.095104:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493844224:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760783.136878:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760783.136891:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760783.136899:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760783.136920:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760783.136947:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493844224:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.137041:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6707, 6707] 00000080:00200000:3.0:1789760783.137053:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.137058:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.137069:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27475968 i_blocks 55305 00000020:00200000:3.0:1789760783.137072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.137091:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.137118:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27475968, count: 4096 00000080:00200000:3.0:1789760783.137124:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27475968, bytes: 4096 00000080:00200000:3.0:1789760783.137132:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.137138:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.137140:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27475968, 27480064) 0 0 00000080:00200000:3.0:1789760783.137144:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.137156:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6708, 6708] 00020000:00200000:3.0:1789760783.137163:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27475968 chunk: [27475968, 27480064) 27480064, 4096 00020000:00200000:3.0:1789760783.137167:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.137180:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6708, 6708] 00000008:00010000:3.0:1789760783.137193:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.137197:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.137207:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27475968, 27480064) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.137219:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27475968=0x1a34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.137263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.137266:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27480064 00000020:00010000:3.0:1789760783.137273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.137277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.137280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.137283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.137285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.137292:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.137298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.137324:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760783.137447:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 27475968<>27480064 00000100:00100000:3.0:1789760783.137491:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.137515:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844352:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.181117:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974027 gen 3 00000008:00010000:2.0:1789760783.181164:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760783.181170:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.181178:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.181199:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760783.181224:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844352:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.181315:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6708, 6708] 00000080:00200000:3.0:1789760783.181327:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.181333:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.181344:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27480064 i_blocks 55305 00000020:00200000:3.0:1789760783.181347:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.181354:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.181381:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27480064, count: 4096 00000080:00200000:3.0:1789760783.181388:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27480064, bytes: 4096 00000080:00200000:3.0:1789760783.181425:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.181432:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.181434:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27480064, 27484160) 0 0 00000080:00200000:3.0:1789760783.181438:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.181451:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6709, 6709] 00020000:00200000:3.0:1789760783.181457:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27480064 chunk: [27480064, 27484160) 27484160, 4096 00020000:00200000:3.0:1789760783.181462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.181476:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6709, 6709] 00000008:00010000:3.0:1789760783.181489:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.181494:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.181509:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27480064, 27484160) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.181519:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27480064=0x1a35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.181561:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.181564:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27484160 00000020:00010000:3.0:1789760783.181570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.181574:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.181576:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.181579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.181581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.181587:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.181592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.181606:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760783.181692:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27480064<>27484160 00000100:00100000:2.0:1789760783.181962:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844480:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.225097:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974028 gen 3 00000008:00010000:2.0:1789760783.225135:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760783.225142:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.225150:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.225170:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000080:00200000:3.0:1789760783.225604:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6709, 6709] 00000080:00200000:3.0:1789760783.225619:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.225625:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.225637:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27484160 i_blocks 55305 00000020:00200000:3.0:1789760783.225641:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.225650:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.225678:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27484160, count: 4096 00000080:00200000:3.0:1789760783.225685:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27484160, bytes: 4096 00000080:00200000:3.0:1789760783.225696:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.225704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.225706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27484160, 27488256) 0 0 00000080:00200000:3.0:1789760783.225710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.225723:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6710, 6710] 00020000:00200000:3.0:1789760783.225731:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27484160 chunk: [27484160, 27488256) 27488256, 4096 00020000:00200000:3.0:1789760783.225735:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.225750:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6710, 6710] 00000008:00010000:3.0:1789760783.225764:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.225769:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.225780:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27484160, 27488256) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.225792:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27484160=0x1a36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.225831:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.225834:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27488256 00000020:00010000:3.0:1789760783.225840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.225844:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.225847:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.225850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.225853:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.225859:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.225864:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760783.226324:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844480:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760783.226363:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:2.0:1789760783.226461:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27484160<>27488256 00000100:00100000:3.0:1789760783.226691:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.227441:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844608:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760783.282421:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760783.282433:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.282442:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.282471:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760783.282501:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844608:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.282883:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6710, 6710] 00000080:00200000:3.0:1789760783.282896:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.282900:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.282912:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27488256 i_blocks 55305 00000020:00200000:3.0:1789760783.282915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.282924:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.282953:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27488256, count: 4096 00000080:00200000:3.0:1789760783.282959:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27488256, bytes: 4096 00000080:00200000:3.0:1789760783.282970:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.282977:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.282979:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27488256, 27492352) 0 0 00000080:00200000:3.0:1789760783.282982:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.282994:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6711, 6711] 00020000:00200000:3.0:1789760783.283000:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27488256 chunk: [27488256, 27492352) 27492352, 4096 00020000:00200000:3.0:1789760783.283004:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.283029:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6711, 6711] 00000008:00010000:3.0:1789760783.283043:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.283048:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.283132:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27488256, 27492352) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.283143:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27488256=0x1a37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.283188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.283190:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27492352 00000020:00010000:3.0:1789760783.283197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.283200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.283203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.283206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.283209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.283216:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.283223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.283238:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760783.283316:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 27488256<>27492352 00000100:00100000:2.0:1789760783.284090:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844736:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.324519:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974029 gen 3 00000100:00100000:2.0:1789760783.324559:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b100 x1876699493844736/t4294974030(4294974030) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760799 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760783.324597:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760783.324603:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.324611:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.324629:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760783.324649:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493844736:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.324788:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6711, 6711] 00000080:00200000:3.0:1789760783.324801:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.324806:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.324819:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27492352 i_blocks 55305 00000020:00200000:3.0:1789760783.324822:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.324832:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.324858:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27492352, count: 4096 00000080:00200000:3.0:1789760783.324865:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27492352, bytes: 4096 00000080:00200000:3.0:1789760783.324873:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.324880:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.324882:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27492352, 27496448) 0 0 00000080:00200000:3.0:1789760783.324886:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.324900:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6712, 6712] 00020000:00200000:3.0:1789760783.324907:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27492352 chunk: [27492352, 27496448) 27496448, 4096 00020000:00200000:3.0:1789760783.324911:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.324926:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6712, 6712] 00000008:00010000:3.0:1789760783.324940:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.324946:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.324955:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27492352, 27496448) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.324966:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27492352=0x1a38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.325008:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.325011:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27496448 00000020:00010000:3.0:1789760783.325027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.325030:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.325033:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.325036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.325039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.325046:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.325052:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.325067:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760783.325156:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 27492352<>27496448 00000100:00100000:2.0:1789760783.325257:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493844864:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.374378:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974031 gen 3 00000008:00010000:2.0:1789760783.374463:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760783.374470:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.374478:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.374499:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000080:00200000:3.0:1789760783.374664:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6712, 6712] 00000100:00100000:2.0:1789760783.374677:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493844864:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.374679:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.374687:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.374698:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27496448 i_blocks 55305 00000020:00200000:3.0:1789760783.374702:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.374711:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.374738:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27496448, count: 4096 00000080:00200000:3.0:1789760783.374743:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27496448, bytes: 4096 00000080:00200000:3.0:1789760783.374750:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.374757:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.374759:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27496448, 27500544) 0 0 00000080:00200000:3.0:1789760783.374763:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.374776:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6713, 6713] 00020000:00200000:3.0:1789760783.374783:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27496448 chunk: [27496448, 27500544) 27500544, 4096 00020000:00200000:3.0:1789760783.374787:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.374801:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6713, 6713] 00000008:00010000:3.0:1789760783.374815:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.374819:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.374829:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27496448, 27500544) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.374839:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27496448=0x1a39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.374881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.374884:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27500544 00000020:00010000:3.0:1789760783.374891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.374894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.374896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.374899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.374901:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.374907:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.374914:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.374927:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760783.375003:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 27496448<>27500544 00000100:00100000:2.0:1789760783.375153:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760783.375167:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493844992:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760783.442365:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760783.442377:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.442388:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.442440:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760783.442468:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493844992:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.442688:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6713, 6713] 00000080:00200000:3.0:1789760783.442700:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.442706:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.442716:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27500544 i_blocks 55305 00000020:00200000:3.0:1789760783.442719:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.442727:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.442754:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27500544, count: 4096 00000080:00200000:3.0:1789760783.442761:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27500544, bytes: 4096 00000080:00200000:3.0:1789760783.442770:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.442777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.442779:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27500544, 27504640) 0 0 00000080:00200000:3.0:1789760783.442783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.442796:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6714, 6714] 00020000:00200000:3.0:1789760783.442803:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27500544 chunk: [27500544, 27504640) 27504640, 4096 00020000:00200000:3.0:1789760783.442807:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.442836:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6714, 6714] 00000008:00010000:3.0:1789760783.442850:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.442855:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.442865:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27500544, 27504640) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.442876:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27500544=0x1a3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.442920:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.442921:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27504640 00000020:00010000:3.0:1789760783.442928:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.442930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.442932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.442934:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.442936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.442940:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.442945:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.442955:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760783.443036:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 27500544<>27504640 00000100:00100000:2.0:1789760783.443201:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845120:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.493114:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974033 gen 3 00000008:00010000:2.0:1789760783.493164:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760783.493170:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.493178:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.493201:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760783.493228:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845120:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.493340:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6714, 6714] 00000080:00200000:3.0:1789760783.493354:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.493360:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.493371:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27504640 i_blocks 55305 00000020:00200000:3.0:1789760783.493375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.493383:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.493434:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27504640, count: 4096 00000080:00200000:3.0:1789760783.493441:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27504640, bytes: 4096 00000080:00200000:3.0:1789760783.493450:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.493457:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.493459:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27504640, 27508736) 0 0 00000080:00200000:3.0:1789760783.493463:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.493477:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6715, 6715] 00020000:00200000:3.0:1789760783.493485:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27504640 chunk: [27504640, 27508736) 27508736, 4096 00020000:00200000:3.0:1789760783.493489:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.493504:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6715, 6715] 00000008:00010000:3.0:1789760783.493518:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.493523:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.493533:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27504640, 27508736) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.493552:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27504640=0x1a3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.493599:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.493602:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27508736 00000020:00010000:3.0:1789760783.493610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.493613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.493616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.493619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.493622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.493629:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.493635:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.493649:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760783.493727:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 27504640<>27508736 00000100:00100000:2.0:1789760783.494076:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845248:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.545205:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974034 gen 3 00000008:00010000:2.0:1789760783.545242:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760783.545248:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.545256:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.545276:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760783.545300:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845248:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.545442:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6715, 6715] 00000080:00200000:3.0:1789760783.545456:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.545462:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.545477:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27508736 i_blocks 55305 00000020:00200000:3.0:1789760783.545480:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.545489:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.545526:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27508736, count: 4096 00000080:00200000:3.0:1789760783.545540:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27508736, bytes: 4096 00000080:00200000:3.0:1789760783.545548:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.545554:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.545556:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27508736, 27512832) 0 0 00000080:00200000:3.0:1789760783.545559:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.545572:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6716, 6716] 00020000:00200000:3.0:1789760783.545580:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27508736 chunk: [27508736, 27512832) 27512832, 4096 00020000:00200000:3.0:1789760783.545583:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.545598:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6716, 6716] 00000008:00010000:3.0:1789760783.545611:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.545615:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.545624:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27508736, 27512832) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.545634:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27508736=0x1a3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.545678:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.545680:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27512832 00000020:00010000:3.0:1789760783.545687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.545690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.545693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.545696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.545699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.545705:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.545710:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.545725:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760783.545916:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 27508736<>27512832 00000100:00100000:2.0:1789760783.546466:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760783.546482:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845376:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760783.599987:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760783.599999:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.600008:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.600035:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760783.600059:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845376:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.600419:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6716, 6716] 00000080:00200000:3.0:1789760783.600432:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.600437:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.600448:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27512832 i_blocks 55305 00000020:00200000:3.0:1789760783.600452:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.600460:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.600487:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27512832, count: 4096 00000080:00200000:3.0:1789760783.600493:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27512832, bytes: 4096 00000080:00200000:3.0:1789760783.600514:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.600522:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.600524:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27512832, 27516928) 0 0 00000080:00200000:3.0:1789760783.600527:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.600549:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6717, 6717] 00020000:00200000:3.0:1789760783.600556:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27512832 chunk: [27512832, 27516928) 27516928, 4096 00020000:00200000:3.0:1789760783.600560:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.600575:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6717, 6717] 00000008:00010000:3.0:1789760783.600591:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.600596:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.600605:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27512832, 27516928) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.600617:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27512832=0x1a3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.600662:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.600665:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27516928 00000020:00010000:3.0:1789760783.600672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.600676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.600679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.600682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.600684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.600690:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.600696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.600711:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760783.600800:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 27512832<>27516928 00000100:00100000:2.0:1789760783.601023:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760783.601041:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845504:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760783.604698:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760783.641595:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974036 gen 3 00000008:00010000:2.0:1789760783.641646:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760783.641653:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.641661:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.641678:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760783.641701:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.641857:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6717, 6717] 00000080:00200000:3.0:1789760783.641870:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.641875:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.641886:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27516928 i_blocks 55305 00000020:00200000:3.0:1789760783.641889:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.641896:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.641922:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27516928, count: 4096 00000080:00200000:3.0:1789760783.641927:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27516928, bytes: 4096 00000080:00200000:3.0:1789760783.641935:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.641943:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.641944:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27516928, 27521024) 0 0 00000080:00200000:3.0:1789760783.641948:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.641961:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6718, 6718] 00020000:00200000:3.0:1789760783.641968:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27516928 chunk: [27516928, 27521024) 27521024, 4096 00020000:00200000:3.0:1789760783.641972:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.641987:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6718, 6718] 00000008:00010000:3.0:1789760783.642001:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.642006:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.642025:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27516928, 27521024) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.642036:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27516928=0x1a3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.642084:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.642086:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27521024 00000020:00010000:3.0:1789760783.642094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.642097:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.642100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.642104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.642107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.642113:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.642118:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.642132:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760783.642212:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 27516928<>27521024 00000100:00100000:2.0:1789760783.642275:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845632:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760783.689569:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974037 gen 3 00000008:00010000:2.0:1789760783.689607:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760783.689614:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.689623:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.689640:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760783.689663:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760783.689870:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6718, 6718] 00000080:00200000:3.0:1789760783.689884:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760783.689889:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760783.689900:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27521024 i_blocks 55305 00000020:00200000:3.0:1789760783.689903:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760783.689911:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760783.689937:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27521024, count: 4096 00000080:00200000:3.0:1789760783.689943:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27521024, bytes: 4096 00000080:00200000:3.0:1789760783.689951:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760783.689958:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760783.689960:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27521024, 27525120) 0 0 00000080:00200000:3.0:1789760783.689964:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760783.689977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6719, 6719] 00020000:00200000:3.0:1789760783.689984:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27521024 chunk: [27521024, 27525120) 27525120, 4096 00020000:00200000:3.0:1789760783.689988:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760783.690001:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6719, 6719] 00000008:00010000:3.0:1789760783.690024:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760783.690029:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760783.690039:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27521024, 27525120) append 0 append lockpos 0 00000080:00200000:3.0:1789760783.690049:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27521024=0x1a3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760783.690091:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760783.690094:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27525120 00000020:00010000:3.0:1789760783.690102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760783.690105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.690108:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760783.690111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760783.690113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760783.690119:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760783.690125:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760783.690139:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760783.690223:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27521024<>27525120 00000100:00100000:2.0:1789760783.690481:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845760:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760783.765168:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974038 gen 3 00000008:00010000:3.0:1789760783.765202:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760783.765208:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.765216:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.765235:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760783.765256:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493845760:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760783.765329:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6719, 6719] 00000080:00200000:2.0:1789760783.765342:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.765347:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.765359:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27525120 i_blocks 55305 00000020:00200000:2.0:1789760783.765362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.765371:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.765416:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27525120, count: 4096 00000080:00200000:2.0:1789760783.765422:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27525120, bytes: 4096 00000080:00200000:2.0:1789760783.765431:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.765438:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.765440:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27525120, 27529216) 0 0 00000080:00200000:2.0:1789760783.765443:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.765454:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6720, 6720] 00020000:00200000:2.0:1789760783.765459:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27525120 chunk: [27525120, 27529216) 27529216, 4096 00020000:00200000:2.0:1789760783.765463:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.765477:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6720, 6720] 00000008:00010000:2.0:1789760783.765489:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.765493:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760783.765502:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27525120, 27529216) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.765511:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27525120=0x1a40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.765563:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.765565:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27529216 00000020:00010000:2.0:1789760783.765572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.765575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.765578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.765580:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.765583:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.765589:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.765594:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.765608:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760783.765682:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27525120<>27529216 00000100:00100000:3.0:1789760783.765844:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845888:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760783.800977:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760783.800987:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.801001:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.801061:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760783.801085:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493845888:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760783.801802:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6720, 6720] 00000080:00200000:2.0:1789760783.801866:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.801872:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.801885:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27529216 i_blocks 55305 00000020:00200000:2.0:1789760783.801889:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.801899:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.801926:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27529216, count: 4096 00000080:00200000:2.0:1789760783.801933:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27529216, bytes: 4096 00000080:00200000:2.0:1789760783.801941:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.801948:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.801950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27529216, 27533312) 0 0 00000080:00200000:2.0:1789760783.801954:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.801967:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6721, 6721] 00020000:00200000:2.0:1789760783.801973:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27529216 chunk: [27529216, 27533312) 27533312, 4096 00020000:00200000:2.0:1789760783.801977:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.801993:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6721, 6721] 00000008:00010000:2.0:1789760783.802006:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.802010:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760783.802032:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27529216, 27533312) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.802044:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27529216=0x1a41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.802090:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.802093:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27533312 00000020:00010000:2.0:1789760783.802099:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.802102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.802104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.802107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.802109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.802115:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.802122:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.802141:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760783.802224:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 27529216<>27533312 00000100:00100000:3.0:1789760783.802294:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493846016:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760783.840190:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974040 gen 3 00000008:00010000:3.0:1789760783.840236:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760783.840243:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.840252:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.840271:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760783.840296:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493846016:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760783.840384:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6721, 6721] 00000080:00200000:2.0:1789760783.840440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.840446:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.840458:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27533312 i_blocks 55305 00000020:00200000:2.0:1789760783.840462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.840471:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.840508:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27533312, count: 4096 00000080:00200000:2.0:1789760783.840515:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27533312, bytes: 4096 00000080:00200000:2.0:1789760783.840523:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.840530:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.840532:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27533312, 27537408) 0 0 00000080:00200000:2.0:1789760783.840536:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.840550:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6722, 6722] 00020000:00200000:2.0:1789760783.840557:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27533312 chunk: [27533312, 27537408) 27537408, 4096 00020000:00200000:2.0:1789760783.840561:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.840576:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6722, 6722] 00000008:00010000:2.0:1789760783.840590:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.840594:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760783.840603:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27533312, 27537408) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.840615:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27533312=0x1a42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.840659:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.840662:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27537408 00000020:00010000:2.0:1789760783.840669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.840672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.840675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.840679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.840681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.840688:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.840694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.840708:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760783.840786:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 27533312<>27537408 00000100:00100000:3.0:1789760783.841012:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846144:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760783.880680:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974041 gen 3 00000008:00010000:3.0:1789760783.880715:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760783.880721:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.880730:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.880792:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760783.880826:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846144:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760783.881202:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6722, 6722] 00000080:00200000:2.0:1789760783.881215:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.881220:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.881232:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27537408 i_blocks 55305 00000020:00200000:2.0:1789760783.881237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.881246:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.881274:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27537408, count: 4096 00000080:00200000:2.0:1789760783.881280:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27537408, bytes: 4096 00000080:00200000:2.0:1789760783.881289:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.881296:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.881298:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27537408, 27541504) 0 0 00000080:00200000:2.0:1789760783.881302:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.881316:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6723, 6723] 00020000:00200000:2.0:1789760783.881323:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27537408 chunk: [27537408, 27541504) 27541504, 4096 00020000:00200000:2.0:1789760783.881327:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.881343:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6723, 6723] 00000008:00010000:2.0:1789760783.881359:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.881364:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760783.881373:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27537408, 27541504) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.881384:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27537408=0x1a43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.882311:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.882316:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27541504 00000020:00010000:2.0:1789760783.882325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.882329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.882332:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.882335:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.882338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.882345:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.882350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.882376:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760783.882734:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 27537408<>27541504 00000100:00100000:2.0:1789760783.882773:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760783.882791:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846272:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760783.926903:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974042 gen 3 00000008:00010000:3.0:1789760783.926938:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760783.926945:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.926954:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.926976:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760783.927001:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846272:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760783.927214:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6723, 6723] 00000080:00200000:0.0:1789760783.927227:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.927232:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.927244:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27541504 i_blocks 55305 00000020:00200000:0.0:1789760783.927247:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.927257:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.927284:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27541504, count: 4096 00000080:00200000:0.0:1789760783.927290:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27541504, bytes: 4096 00000080:00200000:0.0:1789760783.927298:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.927305:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.927307:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27541504, 27545600) 0 0 00000080:00200000:0.0:1789760783.927312:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.927325:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6724, 6724] 00020000:00200000:0.0:1789760783.927332:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27541504 chunk: [27541504, 27545600) 27545600, 4096 00020000:00200000:0.0:1789760783.927337:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.927352:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6724, 6724] 00000008:00010000:0.0:1789760783.927366:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.927371:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760783.927380:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27541504, 27545600) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.927391:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27541504=0x1a44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.927470:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.927472:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27545600 00000020:00010000:0.0:1789760783.927479:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.927483:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.927485:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.927488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.927491:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.927572:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.927578:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.927594:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760783.927670:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 27541504<>27545600 00000100:00100000:0.0:1789760783.927713:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760783.927729:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493846400:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760783.968147:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760783.968157:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760783.968165:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760783.968236:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760783.968264:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493846400:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760783.968329:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6724, 6724] 00000080:00200000:1.0:1789760783.968340:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760783.968345:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760783.968356:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27545600 i_blocks 55305 00000020:00200000:1.0:1789760783.968359:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760783.968368:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760783.968499:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27545600, count: 4096 00000080:00200000:1.0:1789760783.968507:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27545600, bytes: 4096 00000080:00200000:1.0:1789760783.968517:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760783.968524:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760783.968526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27545600, 27549696) 0 0 00000080:00200000:1.0:1789760783.968530:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760783.968554:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6725, 6725] 00020000:00200000:1.0:1789760783.968562:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27545600 chunk: [27545600, 27549696) 27549696, 4096 00020000:00200000:1.0:1789760783.968566:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.968583:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6725, 6725] 00000008:00010000:1.0:1789760783.968598:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760783.968602:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760783.968612:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27545600, 27549696) append 0 append lockpos 0 00000080:00200000:1.0:1789760783.968623:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27545600=0x1a45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760783.968666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760783.968669:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27549696 00000020:00010000:1.0:1789760783.968676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760783.968679:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.968682:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.968685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760783.968687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760783.968695:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760783.968700:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760783.968713:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760783.968789:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 27545600<>27549696 00000100:00100000:1.0:1789760783.968843:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760783.968857:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846528:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760784.014521:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974044 gen 3 00000008:00010000:1.0:1789760784.017232:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:1.0:1789760784.017243:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.017254:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.017370:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:1.0:1789760784.017428:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846528:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760784.017679:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6725, 6725] 00000080:00200000:3.0:1789760784.017694:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760784.017700:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760784.017713:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27549696 i_blocks 55305 00000020:00200000:3.0:1789760784.017717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760784.017726:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760784.017756:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27549696, count: 4096 00000080:00200000:3.0:1789760784.017762:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27549696, bytes: 4096 00000080:00200000:3.0:1789760784.017775:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760784.017783:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760784.017785:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27549696, 27553792) 0 0 00000080:00200000:3.0:1789760784.017789:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760784.017805:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6726, 6726] 00020000:00200000:3.0:1789760784.017812:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27549696 chunk: [27549696, 27553792) 27553792, 4096 00020000:00200000:3.0:1789760784.017817:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760784.017834:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6726, 6726] 00000008:00010000:3.0:1789760784.017852:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760784.017858:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760784.017869:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27549696, 27553792) append 0 append lockpos 0 00000080:00200000:3.0:1789760784.017884:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27549696=0x1a46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760784.017934:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760784.017937:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27553792 00000020:00010000:3.0:1789760784.017943:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760784.017946:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.017948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.017951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760784.017953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760784.017959:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760784.017965:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760784.017982:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760784.018198:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27549696<>27553792 00000100:00100000:2.0:1789760784.018272:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846656:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.063676:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974045 gen 3 00000008:00010000:3.0:1789760784.063711:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760784.063717:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.063725:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.063749:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760784.063775:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493846656:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760784.063886:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6726, 6726] 00000080:00200000:1.0:1789760784.063899:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.063905:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.063917:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27553792 i_blocks 55305 00000020:00200000:1.0:1789760784.063921:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.063928:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.063953:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27553792, count: 4096 00000080:00200000:1.0:1789760784.063959:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27553792, bytes: 4096 00000080:00200000:1.0:1789760784.063966:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.063973:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.063974:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27553792, 27557888) 0 0 00000080:00200000:1.0:1789760784.063978:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.063991:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6727, 6727] 00020000:00200000:1.0:1789760784.063997:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27553792 chunk: [27553792, 27557888) 27557888, 4096 00020000:00200000:1.0:1789760784.064000:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.064026:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6727, 6727] 00000008:00010000:1.0:1789760784.064040:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.064045:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760784.064055:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27553792, 27557888) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.064065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27553792=0x1a47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.064112:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.064115:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27557888 00000020:00010000:1.0:1789760784.064122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.064126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.064129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.064132:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.064135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.064143:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.064149:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.064164:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760784.064244:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 27553792<>27557888 00000100:00100000:0.0:1789760784.064323:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846784:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760784.110454:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974046 gen 3 00000008:00010000:0.0:1789760784.110492:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760784.110509:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.110518:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.110539:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760784.110566:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846784:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760784.110850:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6727, 6727] 00000080:00200000:1.0:1789760784.110863:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.110869:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.110881:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27557888 i_blocks 55305 00000020:00200000:1.0:1789760784.110884:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.110893:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.110918:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27557888, count: 4096 00000080:00200000:1.0:1789760784.110924:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27557888, bytes: 4096 00000080:00200000:1.0:1789760784.110932:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.110939:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.110942:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27557888, 27561984) 0 0 00000080:00200000:1.0:1789760784.110946:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.110959:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6728, 6728] 00020000:00200000:1.0:1789760784.110966:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27557888 chunk: [27557888, 27561984) 27561984, 4096 00020000:00200000:1.0:1789760784.110971:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.110986:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6728, 6728] 00000008:00010000:1.0:1789760784.110999:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.111004:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760784.111024:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27557888, 27561984) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.111035:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27557888=0x1a48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.111093:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.111096:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27561984 00000020:00010000:1.0:1789760784.111103:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.111106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.111109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.111112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.111115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.111124:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.111129:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.111143:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760784.111225:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 27557888<>27561984 00000100:00100000:0.0:1789760784.111382:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.111443:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846912:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760784.157574:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974047 gen 3 00000008:00010000:0.0:1789760784.157613:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760784.157620:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.157628:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.157651:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760784.157680:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493846912:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760784.157838:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6728, 6728] 00000080:00200000:1.0:1789760784.157850:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.157855:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.157868:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27561984 i_blocks 55305 00000020:00200000:1.0:1789760784.157872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.157881:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.157909:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27561984, count: 4096 00000080:00200000:1.0:1789760784.157916:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27561984, bytes: 4096 00000080:00200000:1.0:1789760784.157925:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.157932:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.157934:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27561984, 27566080) 0 0 00000080:00200000:1.0:1789760784.157938:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.157950:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6729, 6729] 00020000:00200000:1.0:1789760784.157957:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27561984 chunk: [27561984, 27566080) 27566080, 4096 00020000:00200000:1.0:1789760784.157961:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.157975:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6729, 6729] 00000008:00010000:1.0:1789760784.157989:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.157994:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760784.158003:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27561984, 27566080) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.158026:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27561984=0x1a49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.158085:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.158087:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27566080 00000020:00010000:1.0:1789760784.158095:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.158098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.158102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.158105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.158107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.158117:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.158123:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.158138:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760784.158217:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 27561984<>27566080 00000100:00100000:1.0:1789760784.158259:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760784.158272:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493847040:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760784.191245:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974048 gen 3 00000008:00010000:1.0:1789760784.191281:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:1.0:1789760784.191287:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.191295:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.191314:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:1.0:1789760784.191341:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493847040:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760784.191534:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6729, 6729] 00000080:00200000:0.0:1789760784.191547:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.191552:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.191563:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27566080 i_blocks 55305 00000020:00200000:0.0:1789760784.191567:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.191575:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.191602:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27566080, count: 4096 00000080:00200000:0.0:1789760784.191608:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27566080, bytes: 4096 00000080:00200000:0.0:1789760784.191616:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.191623:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.191625:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27566080, 27570176) 0 0 00000080:00200000:0.0:1789760784.191629:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.191641:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6730, 6730] 00020000:00200000:0.0:1789760784.191647:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27566080 chunk: [27566080, 27570176) 27570176, 4096 00020000:00200000:0.0:1789760784.191651:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.191664:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6730, 6730] 00000008:00010000:0.0:1789760784.191676:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.191681:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760784.191689:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27566080, 27570176) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.191701:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27566080=0x1a4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.191747:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.191750:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27570176 00000020:00010000:0.0:1789760784.191757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.191760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.191763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.191766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.191769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.191775:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.191780:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.191794:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760784.191868:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 27566080<>27570176 00000100:00100000:0.0:1789760784.191906:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.191919:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493847168:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760784.237544:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974049 gen 3 00000008:00010000:0.0:1789760784.237581:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760784.237587:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.237595:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.237663:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760784.237689:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493847168:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760784.238893:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6730, 6730] 00000080:00200000:3.0:1789760784.238910:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760784.238924:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760784.238937:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27570176 i_blocks 55305 00000020:00200000:3.0:1789760784.238941:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760784.238951:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760784.238978:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27570176, count: 4096 00000080:00200000:3.0:1789760784.238983:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27570176, bytes: 4096 00000080:00200000:3.0:1789760784.238994:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760784.239001:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760784.239003:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27570176, 27574272) 0 0 00000080:00200000:3.0:1789760784.239007:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760784.239185:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6731, 6731] 00020000:00200000:3.0:1789760784.239192:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27570176 chunk: [27570176, 27574272) 27574272, 4096 00020000:00200000:3.0:1789760784.239196:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760784.239213:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6731, 6731] 00000008:00010000:3.0:1789760784.239226:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760784.239231:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760784.239240:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27570176, 27574272) append 0 append lockpos 0 00000080:00200000:3.0:1789760784.239251:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27570176=0x1a4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760784.239312:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760784.239314:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27574272 00000020:00010000:3.0:1789760784.239321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760784.239324:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.239327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760784.239330:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760784.239333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760784.239339:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760784.239345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760784.239500:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760784.239584:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27570176<>27574272 00000100:00100000:3.0:1789760784.239633:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.239652:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847296:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.276599:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974050 gen 3 00000008:00010000:3.0:1789760784.276632:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760784.276638:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.276645:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.276657:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760784.276681:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847296:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760784.277320:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6731, 6731] 00000080:00200000:0.0:1789760784.277335:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.277341:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760784.277352:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27574272 i_blocks 55305 00000020:00200000:0.0:1789760784.277355:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760784.277364:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760784.277390:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27574272, count: 4096 00000080:00200000:0.0:1789760784.277440:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27574272, bytes: 4096 00000080:00200000:0.0:1789760784.277449:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760784.277456:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760784.277457:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27574272, 27578368) 0 0 00000080:00200000:0.0:1789760784.277462:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760784.277474:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6732, 6732] 00020000:00200000:0.0:1789760784.277481:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27574272 chunk: [27574272, 27578368) 27578368, 4096 00020000:00200000:0.0:1789760784.277485:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760784.277510:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6732, 6732] 00000008:00010000:0.0:1789760784.277524:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760784.277529:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760784.277537:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27574272, 27578368) append 0 append lockpos 0 00000080:00200000:0.0:1789760784.277549:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27574272=0x1a4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760784.277595:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760784.277598:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27578368 00000020:00010000:0.0:1789760784.277605:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760784.277609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.277611:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f976c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760784.277615:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760784.277618:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760784.277624:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760784.277630:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760784.277645:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760784.277870:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 27574272<>27578368 00000100:00100000:1.0:1789760784.277959:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493847424:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760784.328955:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974051 gen 3 00000008:00010000:1.0:1789760784.328988:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760784.328994:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760784.329002:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760784.329090:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760784.329117:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493847424:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760784.329771:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6732, 6732] 00000080:00200000:0.0:1789760784.329785:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760784.329790:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.329929:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27578368 i_blocks 55305 00000020:00200000:1.0:1789760784.329933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.329942:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.329968:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27578368, count: 4096 00000080:00200000:1.0:1789760784.329974:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27578368, bytes: 4096 00000080:00200000:1.0:1789760784.329982:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.329989:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.329991:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27578368, 27582464) 0 0 00000080:00200000:1.0:1789760784.329995:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.330007:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6733, 6733] 00020000:00200000:1.0:1789760784.330131:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27578368 chunk: [27578368, 27582464) 27582464, 4096 00020000:00200000:1.0:1789760784.330136:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.330154:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6733, 6733] 00000008:00010000:1.0:1789760784.330168:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.330172:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760784.330180:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27578368, 27582464) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.330191:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27578368=0x1a4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.330238:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.330241:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27582464 00000020:00010000:1.0:1789760784.330247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.330250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.330254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.330257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.330259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.330268:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.330273:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.330428:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760784.330571:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 27578368<>27582464 00000100:00100000:0.0:1789760784.330636:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493847552:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760784.399907:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974052 gen 3 00000008:00010000:0.0:1789760784.399943:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760784.399949:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.399957:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.399976:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760784.400002:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493847552:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.400115:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6733, 6733] 00000080:00200000:2.0:1789760784.400128:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.400133:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.400146:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27582464 i_blocks 55305 00000020:00200000:2.0:1789760784.400150:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.400158:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.400189:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27582464, count: 4096 00000080:00200000:2.0:1789760784.400195:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27582464, bytes: 4096 00000080:00200000:2.0:1789760784.400204:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.400211:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.400214:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27582464, 27586560) 0 0 00000080:00200000:2.0:1789760784.400218:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.400231:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6734, 6734] 00020000:00200000:2.0:1789760784.400237:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27582464 chunk: [27582464, 27586560) 27586560, 4096 00020000:00200000:2.0:1789760784.400241:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.400257:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6734, 6734] 00000008:00010000:2.0:1789760784.400270:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.400275:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.400284:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27582464, 27586560) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.400294:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27582464=0x1a4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.400338:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.400341:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27586560 00000020:00010000:2.0:1789760784.400348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.400352:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.400355:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.400358:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.400361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.400366:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.400372:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.400386:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760784.400513:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27582464<>27586560 00000100:00100000:2.0:1789760784.400549:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760784.400563:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493847680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.458431:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974053 gen 3 00000008:00010000:3.0:1789760784.458472:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760784.458479:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.458488:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.458516:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760784.458541:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493847680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.458677:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6734, 6734] 00000080:00200000:2.0:1789760784.458690:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.458695:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.458708:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27586560 i_blocks 55305 00000020:00200000:2.0:1789760784.458711:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.458719:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.458747:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27586560, count: 4096 00000080:00200000:2.0:1789760784.458753:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27586560, bytes: 4096 00000080:00200000:2.0:1789760784.458760:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.458768:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.458770:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27586560, 27590656) 0 0 00000080:00200000:2.0:1789760784.458774:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.458787:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6735, 6735] 00020000:00200000:2.0:1789760784.458795:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27586560 chunk: [27586560, 27590656) 27590656, 4096 00020000:00200000:2.0:1789760784.458798:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.458812:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6735, 6735] 00000008:00010000:2.0:1789760784.458826:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.458831:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.458840:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27586560, 27590656) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.458851:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27586560=0x1a4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.458893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.458896:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27590656 00000020:00010000:2.0:1789760784.458902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.458906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.458909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.458911:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.458914:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.458920:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.458926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.458941:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760784.459025:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27586560<>27590656 00000100:00100000:3.0:1789760784.459468:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.459487:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.505512:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974054 gen 3 00000008:00010000:3.0:1789760784.505549:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760784.505555:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.505564:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.505582:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760784.505610:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.505835:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6735, 6735] 00000080:00200000:2.0:1789760784.505849:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.505855:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.505867:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27590656 i_blocks 55305 00000020:00200000:2.0:1789760784.505871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.505881:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.505909:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27590656, count: 4096 00000080:00200000:2.0:1789760784.505915:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27590656, bytes: 4096 00000080:00200000:2.0:1789760784.505923:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.506112:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.506114:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27590656, 27594752) 0 0 00000080:00200000:2.0:1789760784.506119:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.506132:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6736, 6736] 00020000:00200000:2.0:1789760784.506138:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27590656 chunk: [27590656, 27594752) 27594752, 4096 00020000:00200000:2.0:1789760784.506143:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.506159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6736, 6736] 00000008:00010000:2.0:1789760784.506172:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.506177:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.506186:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27590656, 27594752) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.506197:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27590656=0x1a50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.506243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.506246:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27594752 00000020:00010000:2.0:1789760784.506253:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.506256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.506260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.506262:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.506265:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.506271:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.506277:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.506290:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760784.506366:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27590656<>27594752 00000100:00100000:3.0:1789760784.509112:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847936:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.549268:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974055 gen 3 00000008:00010000:3.0:1789760784.549304:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760784.549311:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.549321:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.549381:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760784.549447:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493847936:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.549648:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6736, 6736] 00000080:00200000:2.0:1789760784.549662:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.549667:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.549679:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27594752 i_blocks 55305 00000020:00200000:2.0:1789760784.549682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.549691:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.549718:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27594752, count: 4096 00000080:00200000:2.0:1789760784.549725:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27594752, bytes: 4096 00000080:00200000:2.0:1789760784.549734:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.549741:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.549743:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27594752, 27598848) 0 0 00000080:00200000:2.0:1789760784.549747:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.549760:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6737, 6737] 00020000:00200000:2.0:1789760784.549767:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27594752 chunk: [27594752, 27598848) 27598848, 4096 00020000:00200000:2.0:1789760784.549771:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.549786:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6737, 6737] 00000008:00010000:2.0:1789760784.549800:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.549805:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.549814:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27594752, 27598848) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.549888:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27594752=0x1a51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.549936:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.549940:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27598848 00000020:00010000:2.0:1789760784.549947:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.549950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.549952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.549955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.549957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.549963:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.549969:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.549983:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760784.550135:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27594752<>27598848 00000100:00100000:3.0:1789760784.550195:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493848064:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.598755:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974056 gen 3 00000008:00010000:3.0:1789760784.598792:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760784.598798:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.598808:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.598825:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760784.598851:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493848064:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.598945:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6737, 6737] 00000080:00200000:2.0:1789760784.598957:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.598962:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.598974:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27598848 i_blocks 55305 00000020:00200000:2.0:1789760784.598977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.598987:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.599023:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27598848, count: 4096 00000080:00200000:2.0:1789760784.599030:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27598848, bytes: 4096 00000080:00200000:2.0:1789760784.599038:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.599045:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.599047:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27598848, 27602944) 0 0 00000080:00200000:2.0:1789760784.599051:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.599073:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6738, 6738] 00020000:00200000:2.0:1789760784.599080:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27598848 chunk: [27598848, 27602944) 27602944, 4096 00020000:00200000:2.0:1789760784.599084:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.599099:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6738, 6738] 00000008:00010000:2.0:1789760784.599112:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.599117:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.599127:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27598848, 27602944) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.599138:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27598848=0x1a52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.599181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.599184:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27602944 00000020:00010000:2.0:1789760784.599191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.599194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.599197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.599200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.599203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.599208:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.599213:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.599228:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760784.599301:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27598848<>27602944 00000100:00100000:3.0:1789760784.599360:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848192:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760784.623081:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760784.644433:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974057 gen 3 00000008:00010000:3.0:1789760784.644473:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760784.644479:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.644499:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.644515:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760784.644539:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848192:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.644682:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6738, 6738] 00000080:00200000:2.0:1789760784.644695:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.644700:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.644714:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27602944 i_blocks 55305 00000020:00200000:2.0:1789760784.644717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.644726:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.644755:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27602944, count: 4096 00000080:00200000:2.0:1789760784.644761:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27602944, bytes: 4096 00000080:00200000:2.0:1789760784.644769:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.644776:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.644778:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27602944, 27607040) 0 0 00000080:00200000:2.0:1789760784.644783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.644798:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6739, 6739] 00020000:00200000:2.0:1789760784.644805:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27602944 chunk: [27602944, 27607040) 27607040, 4096 00020000:00200000:2.0:1789760784.644808:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.644835:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6739, 6739] 00000008:00010000:2.0:1789760784.644849:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.644854:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.644862:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27602944, 27607040) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.644874:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27602944=0x1a53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.644917:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.644920:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27607040 00000020:00010000:2.0:1789760784.644927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.644930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.644933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.644936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.644939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.644945:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.644951:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.644965:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760784.645054:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27602944<>27607040 00000100:00100000:3.0:1789760784.645205:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.645220:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848320:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.685805:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974058 gen 3 00000008:00010000:3.0:1789760784.685851:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760784.685858:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.685868:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.685885:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760784.685911:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848320:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.686157:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6739, 6739] 00000080:00200000:2.0:1789760784.686169:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.686174:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.686187:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27607040 i_blocks 55305 00000020:00200000:2.0:1789760784.686190:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.686199:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.686226:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27607040, count: 4096 00000080:00200000:2.0:1789760784.686233:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27607040, bytes: 4096 00000080:00200000:2.0:1789760784.686241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.686248:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.686250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27607040, 27611136) 0 0 00000080:00200000:2.0:1789760784.686255:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.686268:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6740, 6740] 00020000:00200000:2.0:1789760784.686275:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27607040 chunk: [27607040, 27611136) 27611136, 4096 00020000:00200000:2.0:1789760784.686280:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.686295:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6740, 6740] 00000008:00010000:2.0:1789760784.686308:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.686313:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.686322:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27607040, 27611136) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.686333:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27607040=0x1a54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.686376:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.686379:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27611136 00000020:00010000:2.0:1789760784.686386:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.686389:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.686431:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.686436:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.686439:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.686445:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.686451:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.686465:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760784.686545:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27607040<>27611136 00000100:00100000:3.0:1789760784.686667:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848448:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.730994:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974059 gen 3 00000008:00010000:3.0:1789760784.731040:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760784.731047:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.731055:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.731080:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760784.731108:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848448:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.731313:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6740, 6740] 00000080:00200000:2.0:1789760784.731326:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.731331:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.731347:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27611136 i_blocks 55305 00000020:00200000:2.0:1789760784.731350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.731359:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.731386:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27611136, count: 4096 00000080:00200000:2.0:1789760784.731391:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27611136, bytes: 4096 00000080:00200000:2.0:1789760784.731429:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.731435:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.731437:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27611136, 27615232) 0 0 00000080:00200000:2.0:1789760784.731441:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.731454:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6741, 6741] 00020000:00200000:2.0:1789760784.731461:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27611136 chunk: [27611136, 27615232) 27615232, 4096 00020000:00200000:2.0:1789760784.731464:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.731480:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6741, 6741] 00000008:00010000:2.0:1789760784.731601:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.731607:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.731616:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27611136, 27615232) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.731627:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27611136=0x1a55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.731669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.731672:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27615232 00000020:00010000:2.0:1789760784.731678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.731681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.731684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.731687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.731689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.731695:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.731700:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.731714:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760784.731791:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 27611136<>27615232 00000100:00100000:3.0:1789760784.731927:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.731943:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848576:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.773970:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974060 gen 3 00000008:00010000:3.0:1789760784.774015:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760784.774022:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.774031:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.774050:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760784.774083:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848576:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.774285:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6741, 6741] 00000080:00200000:2.0:1789760784.774298:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.774304:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.774318:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27615232 i_blocks 55305 00000020:00200000:2.0:1789760784.774323:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.774335:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.774367:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27615232, count: 4096 00000080:00200000:2.0:1789760784.774374:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27615232, bytes: 4096 00000080:00200000:2.0:1789760784.774382:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.774390:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.774423:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27615232, 27619328) 0 0 00000080:00200000:2.0:1789760784.774429:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.774444:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6742, 6742] 00020000:00200000:2.0:1789760784.774453:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27615232 chunk: [27615232, 27619328) 27619328, 4096 00020000:00200000:2.0:1789760784.774457:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.774477:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6742, 6742] 00000008:00010000:2.0:1789760784.774584:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.774589:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.774599:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27615232, 27619328) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.774610:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27615232=0x1a56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.774655:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.774658:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27619328 00000020:00010000:2.0:1789760784.774665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.774668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.774671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.774674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.774677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.774683:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.774688:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.774704:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760784.774788:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 27615232<>27619328 00000100:00100000:3.0:1789760784.775044:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848704:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.812752:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974061 gen 3 00000008:00010000:3.0:1789760784.812787:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760784.812793:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.812802:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.812829:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760784.812854:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848704:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.813004:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6742, 6742] 00000080:00200000:2.0:1789760784.813025:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.813030:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.813042:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27619328 i_blocks 55305 00000020:00200000:2.0:1789760784.813046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.813055:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.813082:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27619328, count: 4096 00000080:00200000:2.0:1789760784.813088:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27619328, bytes: 4096 00000080:00200000:2.0:1789760784.813095:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.813102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.813104:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27619328, 27623424) 0 0 00000080:00200000:2.0:1789760784.813108:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.813121:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6743, 6743] 00020000:00200000:2.0:1789760784.813127:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27619328 chunk: [27619328, 27623424) 27623424, 4096 00020000:00200000:2.0:1789760784.813131:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.813146:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6743, 6743] 00000008:00010000:2.0:1789760784.813158:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.813162:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.813171:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27619328, 27623424) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.813181:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27619328=0x1a57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.813221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.813224:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27623424 00000020:00010000:2.0:1789760784.813230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.813232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.813235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.813238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.813240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.813246:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.813251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.813263:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760784.813336:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27619328<>27623424 00000100:00100000:3.0:1789760784.813431:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493848832:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.850621:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974062 gen 3 00000008:00010000:3.0:1789760784.850656:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760784.850663:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.850672:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.850690:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760784.850714:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493848832:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.850873:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6743, 6743] 00000080:00200000:2.0:1789760784.850887:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.850891:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.850902:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27623424 i_blocks 55305 00000020:00200000:2.0:1789760784.850905:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.850913:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.850939:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27623424, count: 4096 00000080:00200000:2.0:1789760784.850945:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27623424, bytes: 4096 00000080:00200000:2.0:1789760784.850953:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.850959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.850961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27623424, 27627520) 0 0 00000080:00200000:2.0:1789760784.850964:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.850977:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6744, 6744] 00020000:00200000:2.0:1789760784.850983:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27623424 chunk: [27623424, 27627520) 27627520, 4096 00020000:00200000:2.0:1789760784.850987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.851002:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6744, 6744] 00000008:00010000:2.0:1789760784.851025:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.851029:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.851039:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27623424, 27627520) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.851049:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27623424=0x1a58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.851104:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.851107:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27627520 00000020:00010000:2.0:1789760784.851115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.851118:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.851121:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.851124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.851126:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.851133:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.851139:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.851153:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760784.851227:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27623424<>27627520 00000100:00100000:3.0:1789760784.851312:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848960:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760784.890201:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760784.890214:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.890224:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.890243:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760784.890270:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493848960:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.890355:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6744, 6744] 00000080:00200000:2.0:1789760784.890368:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.890373:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.890385:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27627520 i_blocks 55305 00000020:00200000:2.0:1789760784.890388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.890440:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.890469:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27627520, count: 4096 00000080:00200000:2.0:1789760784.890476:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27627520, bytes: 4096 00000080:00200000:2.0:1789760784.890484:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.890502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.890504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27627520, 27631616) 0 0 00000080:00200000:2.0:1789760784.890508:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.890521:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6745, 6745] 00020000:00200000:2.0:1789760784.890528:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27627520 chunk: [27627520, 27631616) 27631616, 4096 00020000:00200000:2.0:1789760784.890531:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.890546:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6745, 6745] 00000008:00010000:2.0:1789760784.890559:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.890563:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.890573:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27627520, 27631616) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.890583:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27627520=0x1a59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.890626:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.890629:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27631616 00000020:00010000:2.0:1789760784.890635:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.890638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.890641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.890644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.890646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.890652:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.890657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.890670:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760784.890753:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27627520<>27631616 00000100:00100000:3.0:1789760784.890853:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.890867:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849088:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760784.927458:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974064 gen 3 00000008:00010000:3.0:1789760784.927517:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760784.927525:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.927535:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.927554:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760784.927582:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849088:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.927717:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6745, 6745] 00000080:00200000:2.0:1789760784.927730:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.927735:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.927747:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27631616 i_blocks 55305 00000020:00200000:2.0:1789760784.927751:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.927760:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.927789:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27631616, count: 4096 00000080:00200000:2.0:1789760784.927795:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27631616, bytes: 4096 00000080:00200000:2.0:1789760784.927803:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.927810:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.927812:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27631616, 27635712) 0 0 00000080:00200000:2.0:1789760784.927816:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.927829:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6746, 6746] 00020000:00200000:2.0:1789760784.927835:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27631616 chunk: [27631616, 27635712) 27635712, 4096 00020000:00200000:2.0:1789760784.927839:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.927854:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6746, 6746] 00000008:00010000:2.0:1789760784.927867:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.927872:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.927881:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27631616, 27635712) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.927891:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27631616=0x1a5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.927942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.927945:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27635712 00000020:00010000:2.0:1789760784.927951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.927955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.927958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.927962:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.927965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.927971:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.927976:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.927991:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760784.928087:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 27631616<>27635712 00000100:00100000:3.0:1789760784.928418:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760784.928431:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493849216:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760784.968680:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760784.968693:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.968703:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.968722:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760784.968747:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493849216:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760784.968842:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6746, 6746] 00000080:00200000:2.0:1789760784.968854:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.968859:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.968871:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27635712 i_blocks 55305 00000020:00200000:2.0:1789760784.968875:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.968884:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.968912:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27635712, count: 4096 00000080:00200000:2.0:1789760784.968918:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27635712, bytes: 4096 00000080:00200000:2.0:1789760784.968925:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.968932:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.969125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27635712, 27639808) 0 0 00000080:00200000:2.0:1789760784.969131:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.969144:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6747, 6747] 00020000:00200000:2.0:1789760784.969151:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27635712 chunk: [27635712, 27639808) 27639808, 4096 00020000:00200000:2.0:1789760784.969155:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.969171:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6747, 6747] 00000008:00010000:2.0:1789760784.969183:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.969188:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760784.969197:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27635712, 27639808) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.969208:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27635712=0x1a5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.969254:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.969258:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27639808 00000020:00010000:2.0:1789760784.969265:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.969268:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.969271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.969275:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.969277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.969284:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.969289:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.969304:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760784.969382:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 27635712<>27639808 00000100:00100000:3.0:1789760784.970594:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493849344:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760785.011848:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974066 gen 3 00000008:00010000:3.0:1789760785.011892:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760785.011899:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.011907:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.012027:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760785.012055:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493849344:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760785.013181:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6747, 6747] 00000080:00200000:2.0:1789760785.013197:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.013203:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.013216:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27639808 i_blocks 55305 00000020:00200000:2.0:1789760785.013221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.013232:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.013261:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27639808, count: 4096 00000080:00200000:2.0:1789760785.013268:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27639808, bytes: 4096 00000080:00200000:2.0:1789760785.013277:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.013285:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.013287:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27639808, 27643904) 0 0 00000080:00200000:2.0:1789760785.013290:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.013303:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6748, 6748] 00020000:00200000:2.0:1789760785.013310:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27639808 chunk: [27639808, 27643904) 27643904, 4096 00020000:00200000:2.0:1789760785.013313:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.013329:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6748, 6748] 00000008:00010000:2.0:1789760785.013343:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.013348:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760785.013357:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27639808, 27643904) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.013369:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27639808=0x1a5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.013437:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.013440:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27643904 00000020:00010000:2.0:1789760785.013447:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.013450:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.013453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.013456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.013459:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.013465:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.013471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.013560:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760785.013643:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27639808<>27643904 00000100:00100000:3.0:1789760785.013716:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849472:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760785.082191:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974067 gen 3 00000008:00010000:3.0:1789760785.082231:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760785.082237:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.082247:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.082267:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760785.082303:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849472:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.082631:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6748, 6748] 00000080:00200000:0.0:1789760785.082644:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.082649:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.082662:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27643904 i_blocks 55305 00000020:00200000:0.0:1789760785.082666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.082675:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.082702:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27643904, count: 4096 00000080:00200000:0.0:1789760785.082708:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27643904, bytes: 4096 00000080:00200000:0.0:1789760785.082717:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.082724:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.082725:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27643904, 27648000) 0 0 00000080:00200000:0.0:1789760785.082729:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.082743:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6749, 6749] 00020000:00200000:0.0:1789760785.082749:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27643904 chunk: [27643904, 27648000) 27648000, 4096 00020000:00200000:0.0:1789760785.082754:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.082770:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6749, 6749] 00000008:00010000:0.0:1789760785.082783:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.082789:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.082799:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27643904, 27648000) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.082812:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27643904=0x1a5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.082859:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.082862:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27648000 00000020:00010000:0.0:1789760785.082869:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.082872:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.082875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.082878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.082880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.082887:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.082893:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.082908:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.083018:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 27643904<>27648000 00000100:00100000:0.0:1789760785.083059:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.083073:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493849600:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.130555:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974068 gen 3 00000008:00010000:0.0:1789760785.130594:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.130602:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.130611:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.130633:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.130793:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493849600:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.131358:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6749, 6749] 00000080:00200000:1.0:1789760785.131371:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.131376:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.131388:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27648000 i_blocks 55305 00000020:00200000:1.0:1789760785.131423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.131434:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.131462:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27648000, count: 4096 00000080:00200000:1.0:1789760785.131469:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27648000, bytes: 4096 00000080:00200000:1.0:1789760785.131479:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.131486:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.131488:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27648000, 27652096) 0 0 00000080:00200000:1.0:1789760785.131492:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.131507:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6750, 6750] 00020000:00200000:1.0:1789760785.131513:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27648000 chunk: [27648000, 27652096) 27652096, 4096 00020000:00200000:1.0:1789760785.131517:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.131532:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6750, 6750] 00000008:00010000:1.0:1789760785.131547:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.131552:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.131561:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27648000, 27652096) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.131573:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27648000=0x1a5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.131621:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.131624:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27652096 00000020:00010000:1.0:1789760785.131630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.131633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.131636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.131639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.131642:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.131651:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.131656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.131673:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760785.131754:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 27648000<>27652096 00000100:00100000:1.0:1789760785.131795:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.131811:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493849728:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.200656:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974069 gen 3 00000008:00010000:1.0:1789760785.200702:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:1.0:1789760785.200708:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.200717:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.200740:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:1.0:1789760785.200765:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493849728:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760785.200871:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6750, 6750] 00000080:00200000:3.0:1789760785.200883:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.200888:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.200899:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27652096 i_blocks 55305 00000020:00200000:3.0:1789760785.200902:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.200910:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.200936:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27652096, count: 4096 00000080:00200000:3.0:1789760785.200941:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27652096, bytes: 4096 00000080:00200000:3.0:1789760785.200948:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.200954:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.200957:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27652096, 27656192) 0 0 00000080:00200000:3.0:1789760785.200960:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.200972:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6751, 6751] 00020000:00200000:3.0:1789760785.200979:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27652096 chunk: [27652096, 27656192) 27656192, 4096 00020000:00200000:3.0:1789760785.200982:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.200996:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6751, 6751] 00000008:00010000:3.0:1789760785.201019:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.201024:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760785.201033:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27652096, 27656192) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.201043:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27652096=0x1a5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.201097:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.201100:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27656192 00000020:00010000:3.0:1789760785.201106:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.201109:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.201112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.201115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.201117:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.201124:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.201130:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.201144:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760785.201219:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27652096<>27656192 00000100:00100000:3.0:1789760785.201258:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.201271:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849856:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760785.277610:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974070 gen 3 00000008:00010000:3.0:1789760785.277647:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760785.277655:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.277663:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.277719:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760785.277742:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849856:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760785.277805:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6751, 6751] 00000080:00200000:2.0:1789760785.277893:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.277899:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.277912:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27656192 i_blocks 55305 00000020:00200000:2.0:1789760785.277915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.277923:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.278034:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27656192, count: 4096 00000080:00200000:2.0:1789760785.278041:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27656192, bytes: 4096 00000080:00200000:2.0:1789760785.278050:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.278122:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.278125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27656192, 27660288) 0 0 00000080:00200000:2.0:1789760785.278129:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.278143:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6752, 6752] 00020000:00200000:2.0:1789760785.278149:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27656192 chunk: [27656192, 27660288) 27660288, 4096 00020000:00200000:2.0:1789760785.278153:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.278168:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6752, 6752] 00000008:00010000:2.0:1789760785.278181:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.278186:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760785.278196:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27656192, 27660288) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.278206:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27656192=0x1a60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.278251:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.278254:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27660288 00000020:00010000:2.0:1789760785.278261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.278264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.278266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.278269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.278272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.278278:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.278284:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.278299:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000008:00100000:2.0:1789760785.278375:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27656192<>27660288 00000100:00100000:3.0:1789760785.278439:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849984:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760785.330482:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000020:00200000:3.0:1789760785.330504:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.330515:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.330524:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b0c0 00000100:00100000:3.0:1789760785.330550:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493849984:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.330939:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6752, 6752] 00000080:00200000:0.0:1789760785.330952:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.330957:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.330968:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27660288 i_blocks 55305 00000020:00200000:0.0:1789760785.330972:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.330981:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.331019:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27660288, count: 4096 00000080:00200000:0.0:1789760785.331025:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27660288, bytes: 4096 00000080:00200000:0.0:1789760785.331033:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.331040:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.331042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27660288, 27664384) 0 0 00000080:00200000:0.0:1789760785.331046:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.331069:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6753, 6753] 00020000:00200000:0.0:1789760785.331077:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27660288 chunk: [27660288, 27664384) 27664384, 4096 00020000:00200000:0.0:1789760785.331081:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.331096:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6753, 6753] 00000008:00010000:0.0:1789760785.331111:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.331117:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.331127:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27660288, 27664384) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.331139:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27660288=0x1a61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.331194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.331197:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27664384 00000020:00010000:0.0:1789760785.331204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.331207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.331210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.331214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.331217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.331224:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.331230:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.331246:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.331330:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 27660288<>27664384 00000100:00100000:0.0:1789760785.331373:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.331389:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850112:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.382687:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974072 gen 3 00000008:00010000:0.0:1789760785.382736:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.382743:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.382749:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.382773:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.382799:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850112:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.383026:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6753, 6753] 00000080:00200000:1.0:1789760785.383038:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.383044:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.383056:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27664384 i_blocks 55305 00000020:00200000:1.0:1789760785.383066:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.383075:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.383102:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27664384, count: 4096 00000080:00200000:1.0:1789760785.383107:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27664384, bytes: 4096 00000080:00200000:1.0:1789760785.383116:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.383123:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.383125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27664384, 27668480) 0 0 00000080:00200000:1.0:1789760785.383128:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.383140:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6754, 6754] 00020000:00200000:1.0:1789760785.383147:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27664384 chunk: [27664384, 27668480) 27668480, 4096 00020000:00200000:1.0:1789760785.383150:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.383165:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6754, 6754] 00000008:00010000:1.0:1789760785.383178:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.383182:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.383190:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27664384, 27668480) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.383201:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27664384=0x1a62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.383243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.383246:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27668480 00000020:00010000:1.0:1789760785.383252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.383255:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.383258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.383261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.383264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.383271:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.383277:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.383293:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760785.383368:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 27664384<>27668480 00000100:00100000:1.0:1789760785.383507:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.383522:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850240:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.431593:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974073 gen 3 00000008:00010000:1.0:1789760785.431631:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:1.0:1789760785.431637:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.431726:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.431747:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:1.0:1789760785.431771:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850240:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.431944:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6754, 6754] 00000080:00200000:0.0:1789760785.431956:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.431961:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.431973:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27668480 i_blocks 55305 00000020:00200000:0.0:1789760785.431976:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.431984:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.432022:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27668480, count: 4096 00000080:00200000:0.0:1789760785.432028:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27668480, bytes: 4096 00000080:00200000:0.0:1789760785.432035:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.432042:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.432045:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27668480, 27672576) 0 0 00000080:00200000:0.0:1789760785.432049:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.432061:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6755, 6755] 00020000:00200000:0.0:1789760785.432069:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27668480 chunk: [27668480, 27672576) 27672576, 4096 00020000:00200000:0.0:1789760785.432073:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.432089:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6755, 6755] 00000008:00010000:0.0:1789760785.432102:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.432107:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.432116:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27668480, 27672576) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.432127:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27668480=0x1a63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.432171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.432174:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27672576 00000020:00010000:0.0:1789760785.432181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.432184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.432187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.432190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.432193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.432199:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.432205:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.432219:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.432299:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 27668480<>27672576 00000100:00100000:0.0:1789760785.432338:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.432351:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850368:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.493426:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974074 gen 3 00000008:00010000:0.0:1789760785.493458:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.493463:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.493470:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.493477:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.493499:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850368:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.493604:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6755, 6755] 00000080:00200000:1.0:1789760785.493617:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.493622:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.493634:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27672576 i_blocks 55305 00000020:00200000:1.0:1789760785.493637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.493721:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.493753:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27672576, count: 4096 00000080:00200000:1.0:1789760785.493759:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27672576, bytes: 4096 00000080:00200000:1.0:1789760785.493767:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.493775:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.493777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27672576, 27676672) 0 0 00000080:00200000:1.0:1789760785.493781:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.493795:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6756, 6756] 00020000:00200000:1.0:1789760785.493802:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27672576 chunk: [27672576, 27676672) 27676672, 4096 00020000:00200000:1.0:1789760785.493806:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.493822:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6756, 6756] 00000008:00010000:1.0:1789760785.493836:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.493840:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.493850:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27672576, 27676672) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.493861:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27672576=0x1a64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.493908:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.493911:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27676672 00000020:00010000:1.0:1789760785.493918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.493932:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.493936:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.493939:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.493942:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.493951:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.493957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.493974:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760785.494084:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 27672576<>27676672 00000100:00100000:0.0:1789760785.494142:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850496:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.531194:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974075 gen 3 00000008:00010000:0.0:1789760785.531235:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760785.531242:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.531250:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.531273:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760785.531300:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850496:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.532026:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6756, 6756] 00000080:00200000:1.0:1789760785.532041:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.532046:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.532123:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27676672 i_blocks 55305 00000020:00200000:1.0:1789760785.532128:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.532138:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.532169:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27676672, count: 4096 00000080:00200000:1.0:1789760785.532176:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27676672, bytes: 4096 00000080:00200000:1.0:1789760785.532186:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.532194:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.532196:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27676672, 27680768) 0 0 00000080:00200000:1.0:1789760785.532200:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.532214:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6757, 6757] 00020000:00200000:1.0:1789760785.532221:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27676672 chunk: [27676672, 27680768) 27680768, 4096 00020000:00200000:1.0:1789760785.532225:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.532240:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6757, 6757] 00000008:00010000:1.0:1789760785.532255:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.532259:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.532269:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27676672, 27680768) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.532281:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27676672=0x1a65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.532421:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.532424:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27680768 00000020:00010000:1.0:1789760785.532431:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.532434:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.532438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.532441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.532444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.532453:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.532459:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.532478:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760785.532629:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 27676672<>27680768 00000100:00100000:1.0:1789760785.532674:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850624:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760785.564280:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760785.564291:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.564299:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.564322:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760785.564348:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850624:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.564522:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6757, 6757] 00000080:00200000:0.0:1789760785.564535:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.564540:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.564552:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27680768 i_blocks 55305 00000020:00200000:0.0:1789760785.564555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.564564:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.564590:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27680768, count: 4096 00000080:00200000:0.0:1789760785.564596:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27680768, bytes: 4096 00000080:00200000:0.0:1789760785.564605:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.564612:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.564613:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27680768, 27684864) 0 0 00000080:00200000:0.0:1789760785.564617:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.564629:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6758, 6758] 00020000:00200000:0.0:1789760785.564637:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27680768 chunk: [27680768, 27684864) 27684864, 4096 00020000:00200000:0.0:1789760785.564641:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.564657:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6758, 6758] 00000008:00010000:0.0:1789760785.564670:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.564675:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.564683:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27680768, 27684864) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.564694:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27680768=0x1a66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.564738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.564740:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27684864 00000020:00010000:0.0:1789760785.564747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.564751:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.564754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.564756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.564759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.564766:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.564771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.564788:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.564936:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 27680768<>27684864 00000100:00100000:0.0:1789760785.564982:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850752:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.601130:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974076 gen 3 00000100:00100000:0.0:1789760785.601169:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ca80 x1876699493850752/t4294974077(4294974077) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760801 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760785.601207:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.601213:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.601221:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.601295:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.601320:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493850752:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.601759:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6758, 6758] 00000080:00200000:1.0:1789760785.601773:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.601779:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.601791:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27684864 i_blocks 55305 00000020:00200000:1.0:1789760785.601795:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.601805:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.601882:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27684864, count: 4096 00000080:00200000:1.0:1789760785.601890:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27684864, bytes: 4096 00000080:00200000:1.0:1789760785.601900:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.601907:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.601909:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27684864, 27688960) 0 0 00000080:00200000:1.0:1789760785.601913:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.601926:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6759, 6759] 00020000:00200000:1.0:1789760785.601934:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27684864 chunk: [27684864, 27688960) 27688960, 4096 00020000:00200000:1.0:1789760785.601938:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.601953:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6759, 6759] 00000008:00010000:1.0:1789760785.601966:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.601971:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.601981:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27684864, 27688960) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.601993:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27684864=0x1a67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.603183:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.603191:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27688960 00000020:00010000:1.0:1789760785.603202:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.603206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.603210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.603214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.603217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.603228:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.603235:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.603267:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760785.603352:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 27684864<>27688960 00000100:00100000:1.0:1789760785.603418:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.637084:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974078 gen 3 00000008:00010000:1.0:1789760785.637136:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:1.0:1789760785.637143:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.637151:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.637176:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:1.0:1789760785.637203:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493850880:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.637383:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6759, 6759] 00000080:00200000:0.0:1789760785.637425:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.637432:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.637444:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27688960 i_blocks 55305 00000020:00200000:0.0:1789760785.637448:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.637457:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.637498:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27688960, count: 4096 00000080:00200000:0.0:1789760785.637504:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27688960, bytes: 4096 00000080:00200000:0.0:1789760785.637513:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.637520:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.637523:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27688960, 27693056) 0 0 00000080:00200000:0.0:1789760785.637526:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.637540:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6760, 6760] 00020000:00200000:0.0:1789760785.637547:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27688960 chunk: [27688960, 27693056) 27693056, 4096 00020000:00200000:0.0:1789760785.637551:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.637567:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6760, 6760] 00000008:00010000:0.0:1789760785.637580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.637585:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.637594:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27688960, 27693056) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.637605:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27688960=0x1a68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.637660:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.637663:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27693056 00000020:00010000:0.0:1789760785.637670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.637673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.637675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.637678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.637681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.637686:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.637692:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.637708:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.637786:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27688960<>27693056 00000100:00100000:0.0:1789760785.637830:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851008:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760785.648022:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760785.668749:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974079 gen 3 00000008:00010000:0.0:1789760785.668784:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.668791:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.668799:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.668829:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.668855:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851008:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.669133:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6760, 6760] 00000080:00200000:1.0:1789760785.669146:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.669151:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.669164:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27693056 i_blocks 55305 00000020:00200000:1.0:1789760785.669167:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.669175:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.669202:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27693056, count: 4096 00000080:00200000:1.0:1789760785.669208:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27693056, bytes: 4096 00000080:00200000:1.0:1789760785.669216:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.669223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.669225:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27693056, 27697152) 0 0 00000080:00200000:1.0:1789760785.669228:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.669240:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6761, 6761] 00020000:00200000:1.0:1789760785.669247:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27693056 chunk: [27693056, 27697152) 27697152, 4096 00020000:00200000:1.0:1789760785.669251:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.669266:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6761, 6761] 00000008:00010000:1.0:1789760785.669280:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.669284:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.669292:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27693056, 27697152) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.669303:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27693056=0x1a69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.669350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.669352:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27697152 00000020:00010000:1.0:1789760785.669358:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.669361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.669364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.669367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.669369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.669376:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.669381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.669762:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760785.669905:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 27693056<>27697152 00000100:00100000:0.0:1789760785.669964:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.670050:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851136:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.714121:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974080 gen 3 00000008:00010000:0.0:1789760785.714164:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760785.714172:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.714182:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.714205:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760785.714239:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851136:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.714263:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6761, 6761] 00000080:00200000:1.0:1789760785.714277:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.714282:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.714294:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27697152 i_blocks 55305 00000020:00200000:1.0:1789760785.714298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.714306:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.714333:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27697152, count: 4096 00000080:00200000:1.0:1789760785.714340:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27697152, bytes: 4096 00000080:00200000:1.0:1789760785.714350:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.714357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.714359:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27697152, 27701248) 0 0 00000080:00200000:1.0:1789760785.714363:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.714375:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6762, 6762] 00020000:00200000:1.0:1789760785.714382:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27697152 chunk: [27697152, 27701248) 27701248, 4096 00020000:00200000:1.0:1789760785.714386:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.714511:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6762, 6762] 00000008:00010000:1.0:1789760785.714541:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.714546:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.714555:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27697152, 27701248) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.714567:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27697152=0x1a6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.714607:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.714609:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27701248 00000020:00010000:1.0:1789760785.714617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.714621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000008:00010000:0.0:1789760785.714622:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00010000:1.0:1789760785.714626:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.714632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.714634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.714643:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.714649:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00100000:0.0:1789760785.714708:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27697152<>27701248 00000100:00100000:1.0:1789760785.714806:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.715034:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851264:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.760453:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974081 gen 3 00000008:00010000:1.0:1789760785.760503:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:1.0:1789760785.760510:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.760518:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.760543:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:1.0:1789760785.760569:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851264:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.760730:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6762, 6762] 00000080:00200000:0.0:1789760785.760744:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.760749:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.760762:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27701248 i_blocks 55305 00000020:00200000:0.0:1789760785.760766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.760774:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.760803:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27701248, count: 4096 00000080:00200000:0.0:1789760785.760809:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27701248, bytes: 4096 00000080:00200000:0.0:1789760785.760830:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.760838:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.760840:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27701248, 27705344) 0 0 00000080:00200000:0.0:1789760785.760844:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.760857:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6763, 6763] 00020000:00200000:0.0:1789760785.760864:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27701248 chunk: [27701248, 27705344) 27705344, 4096 00020000:00200000:0.0:1789760785.760867:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.760883:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6763, 6763] 00000008:00010000:0.0:1789760785.760897:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.760902:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.760912:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27701248, 27705344) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.760923:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27701248=0x1a6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.760967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.760970:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27705344 00000020:00010000:0.0:1789760785.760977:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.760981:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.760984:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.760987:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.760990:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.760996:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.761012:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.761028:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.761111:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27701248<>27705344 00000100:00100000:0.0:1789760785.761146:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.761161:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851392:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.793679:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974082 gen 3 00000008:00010000:0.0:1789760785.793716:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.793722:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.793731:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.793814:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.793853:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851392:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.794466:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6763, 6763] 00000080:00200000:1.0:1789760785.794548:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.794554:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.794568:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27705344 i_blocks 55305 00000020:00200000:1.0:1789760785.794571:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.794580:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.794610:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27705344, count: 4096 00000080:00200000:1.0:1789760785.794617:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27705344, bytes: 4096 00000080:00200000:1.0:1789760785.794626:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.794633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.794635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27705344, 27709440) 0 0 00000080:00200000:1.0:1789760785.794639:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.794653:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6764, 6764] 00020000:00200000:1.0:1789760785.794661:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27705344 chunk: [27705344, 27709440) 27709440, 4096 00020000:00200000:1.0:1789760785.794664:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.794679:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6764, 6764] 00000008:00010000:1.0:1789760785.794691:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.794695:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.794704:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27705344, 27709440) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.794715:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27705344=0x1a6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.794760:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.794762:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27709440 00000020:00010000:1.0:1789760785.794769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.794772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.794775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.794778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.794780:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.794789:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.794793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.794877:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760785.794960:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 27705344<>27709440 00000100:00100000:0.0:1789760785.795037:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851520:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.841241:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974083 gen 3 00000008:00010000:0.0:1789760785.841278:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760785.841285:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.841295:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.841315:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760785.841341:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851520:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.841362:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6764, 6764] 00000080:00200000:1.0:1789760785.841375:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.841380:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.841390:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27709440 i_blocks 55305 00000020:00200000:1.0:1789760785.841420:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.841429:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.841459:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27709440, count: 4096 00000080:00200000:1.0:1789760785.841465:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27709440, bytes: 4096 00000080:00200000:1.0:1789760785.841472:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.841480:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.841482:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27709440, 27713536) 0 0 00000080:00200000:1.0:1789760785.841486:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.841498:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6765, 6765] 00020000:00200000:1.0:1789760785.841504:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27709440 chunk: [27709440, 27713536) 27713536, 4096 00020000:00200000:1.0:1789760785.841508:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.841523:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6765, 6765] 00000008:00010000:1.0:1789760785.841537:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.841541:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.841550:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27709440, 27713536) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.841561:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27709440=0x1a6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.841604:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.841607:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27713536 00000020:00010000:1.0:1789760785.841614:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.841617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.841619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.841622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.841625:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.841634:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.841639:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.841730:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760785.841798:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 27709440<>27713536 00000100:00100000:1.0:1789760785.841830:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851648:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.882594:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974084 gen 3 00000008:00010000:1.0:1789760785.882632:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760785.882639:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.882646:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.882668:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760785.882692:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851648:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.882790:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6765, 6765] 00000080:00200000:0.0:1789760785.882802:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.882808:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.882818:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27713536 i_blocks 55305 00000020:00200000:0.0:1789760785.882821:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.882830:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.882858:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27713536, count: 4096 00000080:00200000:0.0:1789760785.882864:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27713536, bytes: 4096 00000080:00200000:0.0:1789760785.882872:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.882880:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.882881:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27713536, 27717632) 0 0 00000080:00200000:0.0:1789760785.882885:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.882898:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6766, 6766] 00020000:00200000:0.0:1789760785.882905:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27713536 chunk: [27713536, 27717632) 27717632, 4096 00020000:00200000:0.0:1789760785.882909:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.883300:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6766, 6766] 00000008:00010000:0.0:1789760785.883316:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.883321:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.883332:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27713536, 27717632) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.883343:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27713536=0x1a6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.883390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.883768:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27717632 00000020:00010000:0.0:1789760785.883778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.883782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.883785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.883788:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.883791:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.883798:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.883804:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.883825:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.883908:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27713536<>27717632 00000100:00100000:0.0:1789760785.883957:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760785.883975:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851776:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760785.923138:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974085 gen 3 00000008:00010000:0.0:1789760785.923185:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760785.923192:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.923201:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.923225:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760785.923252:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493851776:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760785.923354:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6766, 6766] 00000080:00200000:1.0:1789760785.923367:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.923371:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.923384:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27717632 i_blocks 55305 00000020:00200000:1.0:1789760785.923388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.923422:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.923453:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27717632, count: 4096 00000080:00200000:1.0:1789760785.923459:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27717632, bytes: 4096 00000080:00200000:1.0:1789760785.923468:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.923475:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.923477:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27717632, 27721728) 0 0 00000080:00200000:1.0:1789760785.923493:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.923507:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6767, 6767] 00020000:00200000:1.0:1789760785.923514:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27717632 chunk: [27717632, 27721728) 27721728, 4096 00020000:00200000:1.0:1789760785.923518:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.923534:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6767, 6767] 00000008:00010000:1.0:1789760785.923550:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.923555:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760785.923564:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27717632, 27721728) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.923575:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27717632=0x1a6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.923619:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.923622:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27721728 00000020:00010000:1.0:1789760785.923627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.923630:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.923633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.923636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.923638:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.923647:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.923652:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.923666:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760785.923745:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 27717632<>27721728 00000100:00100000:1.0:1789760785.923778:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760785.923791:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851904:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760785.962607:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974086 gen 3 00000008:00010000:1.0:1789760785.962645:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760785.962652:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.962660:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.962683:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760785.962709:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493851904:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760785.962863:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6767, 6767] 00000080:00200000:0.0:1789760785.962876:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.962881:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.962891:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27721728 i_blocks 55305 00000020:00200000:0.0:1789760785.962895:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.962904:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.962930:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27721728, count: 4096 00000080:00200000:0.0:1789760785.962936:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27721728, bytes: 4096 00000080:00200000:0.0:1789760785.962944:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.962951:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.962953:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27721728, 27725824) 0 0 00000080:00200000:0.0:1789760785.962957:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.962969:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6768, 6768] 00020000:00200000:0.0:1789760785.962975:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27721728 chunk: [27721728, 27725824) 27725824, 4096 00020000:00200000:0.0:1789760785.962979:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.962993:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6768, 6768] 00000008:00010000:0.0:1789760785.963016:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.963021:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760785.963389:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27721728, 27725824) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.963446:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27721728=0x1a70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.963491:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.963494:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27725824 00000020:00010000:0.0:1789760785.963501:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.963504:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.963507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.963510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.963512:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.963519:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.963524:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.963542:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760785.963633:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27721728<>27725824 00000100:00100000:0.0:1789760785.963672:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760785.963688:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852032:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760786.037293:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974087 gen 3 00000008:00010000:0.0:1789760786.037331:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760786.037337:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.037345:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.037370:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760786.037425:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852032:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760786.037983:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6768, 6768] 00000080:00200000:1.0:1789760786.038006:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760786.038012:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760786.038023:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27725824 i_blocks 55305 00000020:00200000:1.0:1789760786.038027:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760786.038037:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760786.038075:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27725824, count: 4096 00000080:00200000:1.0:1789760786.038080:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27725824, bytes: 4096 00000080:00200000:1.0:1789760786.038088:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760786.038095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760786.038097:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27725824, 27729920) 0 0 00000080:00200000:1.0:1789760786.038101:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760786.038114:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6769, 6769] 00020000:00200000:1.0:1789760786.038120:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27725824 chunk: [27725824, 27729920) 27729920, 4096 00020000:00200000:1.0:1789760786.038125:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760786.038139:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6769, 6769] 00000008:00010000:1.0:1789760786.038152:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760786.038157:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760786.038165:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27725824, 27729920) append 0 append lockpos 0 00000080:00200000:1.0:1789760786.038177:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27725824=0x1a71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760786.038223:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760786.038226:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27729920 00000020:00010000:1.0:1789760786.038232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760786.038235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760786.038237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760786.038240:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760786.038243:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760786.038251:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760786.038256:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760786.038272:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760786.038356:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 27725824<>27729920 00000100:00100000:1.0:1789760786.038427:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760786.038443:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493852160:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.103490:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974088 gen 3 00000008:00010000:1.0:1789760786.103526:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760786.103532:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.103541:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.103564:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000080:00200000:0.0:1789760786.103700:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6769, 6769] 00000080:00200000:0.0:1789760786.103714:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.103719:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.103730:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27729920 i_blocks 55305 00000020:00200000:0.0:1789760786.103734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.103742:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.103769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27729920, count: 4096 00000080:00200000:0.0:1789760786.103776:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27729920, bytes: 4096 00000080:00200000:0.0:1789760786.103784:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.103791:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.103794:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27729920, 27734016) 0 0 00000080:00200000:0.0:1789760786.103798:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.103812:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6770, 6770] 00020000:00200000:0.0:1789760786.103819:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27729920 chunk: [27729920, 27734016) 27734016, 4096 00020000:00200000:0.0:1789760786.103823:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.103838:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6770, 6770] 00000008:00010000:0.0:1789760786.103851:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.103856:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.103865:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27729920, 27734016) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.103876:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27729920=0x1a72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.103937:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.103939:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27734016 00000020:00010000:0.0:1789760786.103945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.103948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.103951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.103954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.103957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.103962:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.103967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.103983:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760786.104064:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27729920<>27734016 00000100:00100000:0.0:1789760786.104088:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760786.104105:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852288:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.104362:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493852160:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760786.186706:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974089 gen 3 00000008:00010000:0.0:1789760786.186876:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760786.186883:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.186892:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.186914:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760786.187057:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852288:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760786.187164:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6770, 6770] 00000080:00200000:1.0:1789760786.187178:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760786.187183:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760786.187195:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27734016 i_blocks 55305 00000020:00200000:1.0:1789760786.187198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760786.187207:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760786.187235:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27734016, count: 4096 00000080:00200000:1.0:1789760786.187242:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27734016, bytes: 4096 00000080:00200000:1.0:1789760786.187250:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760786.187258:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760786.187259:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27734016, 27738112) 0 0 00000080:00200000:1.0:1789760786.187263:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760786.187276:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6771, 6771] 00020000:00200000:1.0:1789760786.187282:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27734016 chunk: [27734016, 27738112) 27738112, 4096 00020000:00200000:1.0:1789760786.187285:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760786.187313:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6771, 6771] 00000008:00010000:1.0:1789760786.187327:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760786.187332:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760786.187340:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27734016, 27738112) append 0 append lockpos 0 00000080:00200000:1.0:1789760786.187350:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27734016=0x1a73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760786.187433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760786.187435:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27738112 00000020:00010000:1.0:1789760786.187443:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:1.0:1789760786.187445:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760786.187448:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760786.187451:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760786.187453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760786.187463:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760786.187469:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760786.187484:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760786.187566:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 27734016<>27738112 00000100:00100000:1.0:1789760786.187857:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760786.187985:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493852416:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.235338:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974090 gen 3 00000008:00010000:1.0:1789760786.235373:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:1.0:1789760786.235379:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.235388:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.235441:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:1.0:1789760786.235465:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493852416:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760786.235579:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6771, 6771] 00000080:00200000:0.0:1789760786.235592:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.235597:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.235608:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27738112 i_blocks 55305 00000020:00200000:0.0:1789760786.235612:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.235620:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.235646:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27738112, count: 4096 00000080:00200000:0.0:1789760786.235652:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27738112, bytes: 4096 00000080:00200000:0.0:1789760786.235660:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.235667:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.235669:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27738112, 27742208) 0 0 00000080:00200000:0.0:1789760786.235672:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.235684:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6772, 6772] 00020000:00200000:0.0:1789760786.235691:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27738112 chunk: [27738112, 27742208) 27742208, 4096 00020000:00200000:0.0:1789760786.235695:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.235711:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6772, 6772] 00000008:00010000:0.0:1789760786.235724:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.235728:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.235736:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27738112, 27742208) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.235747:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27738112=0x1a74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.235788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.235791:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27742208 00000020:00010000:0.0:1789760786.235796:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.235799:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.235802:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.235805:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.235807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.235813:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.235830:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.235844:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760786.235916:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c700 - object 0x240000400:5 offset 27738112<>27742208 00000100:00100000:0.0:1789760786.235952:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760786.235964:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852544:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760786.283168:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760786.283179:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.283188:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.283212:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:0.0:1789760786.283239:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493852544:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.283418:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6772, 6772] 00000080:00200000:2.0:1789760786.283431:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.283436:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.283446:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27742208 i_blocks 55305 00000020:00200000:2.0:1789760786.283449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.283457:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.283584:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27742208, count: 4096 00000080:00200000:2.0:1789760786.283592:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27742208, bytes: 4096 00000080:00200000:2.0:1789760786.283601:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.283608:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.283610:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27742208, 27746304) 0 0 00000080:00200000:2.0:1789760786.283615:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.283628:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6773, 6773] 00020000:00200000:2.0:1789760786.283634:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27742208 chunk: [27742208, 27746304) 27746304, 4096 00020000:00200000:2.0:1789760786.283637:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.283662:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6773, 6773] 00000008:00010000:2.0:1789760786.283675:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.283680:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.283689:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27742208, 27746304) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.283699:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27742208=0x1a75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.283739:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.283742:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27746304 00000020:00010000:2.0:1789760786.283747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.283750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.283753:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.283756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.283758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.283764:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.283769:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.283783:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760786.283874:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27742208<>27746304 00000100:00100000:3.0:1789760786.283961:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493852672:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.335070:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974092 gen 3 00000008:00010000:3.0:1789760786.335122:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760786.335129:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.335137:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.335151:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760786.335178:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493852672:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.335179:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6773, 6773] 00000080:00200000:2.0:1789760786.335192:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.335198:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.335209:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27746304 i_blocks 55305 00000020:00200000:2.0:1789760786.335212:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.335221:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.335248:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27746304, count: 4096 00000080:00200000:2.0:1789760786.335254:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27746304, bytes: 4096 00000080:00200000:2.0:1789760786.335261:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.335268:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.335270:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27746304, 27750400) 0 0 00000080:00200000:2.0:1789760786.335274:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.335286:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6774, 6774] 00020000:00200000:2.0:1789760786.335305:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27746304 chunk: [27746304, 27750400) 27750400, 4096 00020000:00200000:2.0:1789760786.335309:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.335382:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6774, 6774] 00000008:00010000:2.0:1789760786.335427:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.335432:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.335441:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27746304, 27750400) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.335451:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27746304=0x1a76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.335558:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.335561:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27750400 00000020:00010000:2.0:1789760786.335569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.335573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.335575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.335579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.335581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.335588:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.335593:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.335607:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760786.335682:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27746304<>27750400 00000100:00100000:3.0:1789760786.335741:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493852800:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.368120:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974093 gen 3 00000008:00010000:3.0:1789760786.368154:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760786.368160:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.368168:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.368189:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760786.368214:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493852800:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.368306:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6774, 6774] 00000080:00200000:2.0:1789760786.368319:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.368324:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.368335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27750400 i_blocks 55305 00000020:00200000:2.0:1789760786.368338:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.368347:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.368373:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27750400, count: 4096 00000080:00200000:2.0:1789760786.368379:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27750400, bytes: 4096 00000080:00200000:2.0:1789760786.368387:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.368423:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.368425:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27750400, 27754496) 0 0 00000080:00200000:2.0:1789760786.368429:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.368440:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6775, 6775] 00020000:00200000:2.0:1789760786.368447:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27750400 chunk: [27750400, 27754496) 27754496, 4096 00020000:00200000:2.0:1789760786.368451:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.368466:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6775, 6775] 00000008:00010000:2.0:1789760786.368490:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.368495:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.368503:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27750400, 27754496) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.368514:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27750400=0x1a77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.368557:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.368561:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27754496 00000020:00010000:2.0:1789760786.368567:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.368570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.368573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.368577:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.368580:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.368586:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.368592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.368605:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760786.368678:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27750400<>27754496 00000100:00100000:3.0:1789760786.368933:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493852928:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.411235:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974094 gen 3 00000008:00010000:3.0:1789760786.411275:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760786.411282:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.411544:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.411604:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760786.411631:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493852928:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.411702:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6775, 6775] 00000080:00200000:2.0:1789760786.411716:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.411721:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.411731:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27754496 i_blocks 55305 00000020:00200000:2.0:1789760786.411735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.411744:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.411769:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27754496, count: 4096 00000080:00200000:2.0:1789760786.411775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27754496, bytes: 4096 00000080:00200000:2.0:1789760786.411783:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.411790:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.411792:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27754496, 27758592) 0 0 00000080:00200000:2.0:1789760786.411796:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.411810:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6776, 6776] 00020000:00200000:2.0:1789760786.411875:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27754496 chunk: [27754496, 27758592) 27758592, 4096 00020000:00200000:2.0:1789760786.411880:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.411897:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6776, 6776] 00000008:00010000:2.0:1789760786.411910:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.411915:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.411924:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27754496, 27758592) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.411934:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27754496=0x1a78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.412053:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.412056:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27758592 00000020:00010000:2.0:1789760786.412063:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.412067:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.412070:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.412073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.412075:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.412081:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.412086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.412099:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760786.412769:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27754496<>27758592 00000100:00100000:3.0:1789760786.412834:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493853056:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.471121:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974095 gen 3 00000008:00010000:3.0:1789760786.471157:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760786.471163:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.471173:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.471196:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760786.471222:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493853056:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760786.471507:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6776, 6776] 00000080:00200000:0.0:1789760786.471519:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.471525:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.471538:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27758592 i_blocks 55305 00000020:00200000:0.0:1789760786.471542:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.471550:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.471577:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27758592, count: 4096 00000080:00200000:0.0:1789760786.471583:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27758592, bytes: 4096 00000080:00200000:0.0:1789760786.471592:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.471599:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.471601:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27758592, 27762688) 0 0 00000080:00200000:0.0:1789760786.471605:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.471619:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6777, 6777] 00020000:00200000:0.0:1789760786.471626:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27758592 chunk: [27758592, 27762688) 27762688, 4096 00020000:00200000:0.0:1789760786.471630:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.471645:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6777, 6777] 00000008:00010000:0.0:1789760786.471660:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.471665:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.471675:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27758592, 27762688) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.471687:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27758592=0x1a79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.471734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.471737:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27762688 00000020:00010000:0.0:1789760786.471745:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.471749:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.471752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.471756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.471760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.471767:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.471773:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.471788:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760786.471871:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 27758592<>27762688 00000100:00100000:1.0:1789760786.471954:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853184:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.504228:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974096 gen 3 00000008:00010000:1.0:1789760786.504265:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760786.504272:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.504435:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.504455:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760786.504481:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853184:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760786.504541:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6777, 6777] 00000080:00200000:0.0:1789760786.504556:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.504561:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.504573:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27762688 i_blocks 55305 00000020:00200000:0.0:1789760786.504576:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.504585:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.504611:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27762688, count: 4096 00000080:00200000:0.0:1789760786.504617:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27762688, bytes: 4096 00000080:00200000:0.0:1789760786.504626:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.504633:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.504635:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27762688, 27766784) 0 0 00000080:00200000:0.0:1789760786.504638:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.504649:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6778, 6778] 00020000:00200000:0.0:1789760786.504656:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27762688 chunk: [27762688, 27766784) 27766784, 4096 00020000:00200000:0.0:1789760786.504659:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.504673:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6778, 6778] 00000008:00010000:0.0:1789760786.504685:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.504689:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.504698:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27762688, 27766784) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.504707:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27762688=0x1a7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.504746:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.504748:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27766784 00000020:00010000:0.0:1789760786.504755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.504758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.504760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.504763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.504766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.504771:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.504777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.504794:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760786.504880:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 27762688<>27766784 00000100:00100000:1.0:1789760786.505166:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760786.505180:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853312:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.553525:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974097 gen 3 00000008:00010000:1.0:1789760786.553577:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760786.553586:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.553597:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.553622:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760786.553651:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853312:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760786.553818:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6778, 6778] 00000080:00200000:0.0:1789760786.553834:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.553840:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.553856:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27766784 i_blocks 55305 00000020:00200000:0.0:1789760786.553861:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.553871:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.553903:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27766784, count: 4096 00000080:00200000:0.0:1789760786.553911:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27766784, bytes: 4096 00000080:00200000:0.0:1789760786.553920:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.553929:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.553931:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27766784, 27770880) 0 0 00000080:00200000:0.0:1789760786.553935:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.553952:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6779, 6779] 00020000:00200000:0.0:1789760786.553961:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27766784 chunk: [27766784, 27770880) 27770880, 4096 00020000:00200000:0.0:1789760786.553965:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.553983:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6779, 6779] 00000008:00010000:0.0:1789760786.554011:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.554017:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.554030:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27766784, 27770880) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.554044:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27766784=0x1a7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.554172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.554175:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27770880 00000020:00010000:0.0:1789760786.554183:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.554187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.554190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.554194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.554198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.554205:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.554213:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.554227:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760786.554739:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 27766784<>27770880 00000100:00100000:1.0:1789760786.555906:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853440:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760786.591932:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974098 gen 3 00000008:00010000:1.0:1789760786.591979:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760786.591988:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.592009:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.592031:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760786.592065:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493853440:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760786.592332:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6779, 6779] 00000080:00200000:0.0:1789760786.592348:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.592354:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.592370:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27770880 i_blocks 55305 00000020:00200000:0.0:1789760786.592374:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.592384:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.592445:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27770880, count: 4096 00000080:00200000:0.0:1789760786.592454:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27770880, bytes: 4096 00000080:00200000:0.0:1789760786.592464:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.592536:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.592540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27770880, 27774976) 0 0 00000080:00200000:0.0:1789760786.592546:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.592563:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6780, 6780] 00020000:00200000:0.0:1789760786.592571:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27770880 chunk: [27770880, 27774976) 27774976, 4096 00020000:00200000:0.0:1789760786.592576:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.592597:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6780, 6780] 00000008:00010000:0.0:1789760786.593005:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.593013:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760786.593028:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27770880, 27774976) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.593043:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27770880=0x1a7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.593115:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.593118:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27774976 00000020:00010000:0.0:1789760786.593127:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.593130:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.593133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.593137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.593141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.593149:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.593156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.593175:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760786.593273:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 27770880<>27774976 00000100:00100000:1.0:1789760786.593354:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493853568:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760786.641344:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760786.641361:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.641373:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.641562:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760786.641592:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493853568:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760786.642136:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6780, 6780] 00000080:00200000:3.0:1789760786.642154:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.642161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.642177:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27774976 i_blocks 55305 00000020:00200000:3.0:1789760786.642181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.642192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.642227:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27774976, count: 4096 00000080:00200000:3.0:1789760786.642234:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27774976, bytes: 4096 00000080:00200000:3.0:1789760786.642246:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.642255:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.642258:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27774976, 27779072) 0 0 00000080:00200000:3.0:1789760786.642263:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.642279:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6781, 6781] 00020000:00200000:3.0:1789760786.642288:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27774976 chunk: [27774976, 27779072) 27779072, 4096 00020000:00200000:3.0:1789760786.642293:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.642313:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6781, 6781] 00000008:00010000:3.0:1789760786.642332:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.642338:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760786.642350:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27774976, 27779072) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.642365:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27774976=0x1a7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.642664:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.642668:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27779072 00000020:00010000:3.0:1789760786.642677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.642680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.642684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.642687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.642691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.642698:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.642705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.642722:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760786.642952:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27774976<>27779072 00000100:00100000:2.0:1789760786.643022:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853696:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760786.668622:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760786.674533:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974100 gen 3 00000008:00010000:3.0:1789760786.674599:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760786.674608:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.674619:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.674642:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760786.674671:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.674807:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6781, 6781] 00000080:00200000:2.0:1789760786.674824:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.674830:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.674844:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27779072 i_blocks 55305 00000020:00200000:2.0:1789760786.674848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.674856:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.674891:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27779072, count: 4096 00000080:00200000:2.0:1789760786.674899:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27779072, bytes: 4096 00000080:00200000:2.0:1789760786.674908:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.674916:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.674918:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27779072, 27783168) 0 0 00000080:00200000:2.0:1789760786.674923:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.674939:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6782, 6782] 00020000:00200000:2.0:1789760786.674946:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27779072 chunk: [27779072, 27783168) 27783168, 4096 00020000:00200000:2.0:1789760786.674950:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.674970:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6782, 6782] 00000008:00010000:2.0:1789760786.674989:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.675006:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.675018:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27779072, 27783168) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.675032:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27779072=0x1a7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.675091:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.675093:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27783168 00000020:00010000:2.0:1789760786.675102:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.675105:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.675108:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.675112:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.675115:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.675122:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.675129:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.675145:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760786.675247:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27779072<>27783168 00000100:00100000:3.0:1789760786.675724:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.675748:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853824:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760786.713514:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760786.713532:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.713544:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.713602:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760786.713632:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853824:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.713700:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6782, 6782] 00000080:00200000:2.0:1789760786.713716:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.713722:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.713738:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27783168 i_blocks 55305 00000020:00200000:2.0:1789760786.713741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.713752:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.713785:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27783168, count: 4096 00000080:00200000:2.0:1789760786.713792:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27783168, bytes: 4096 00000080:00200000:2.0:1789760786.713802:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.713811:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.713813:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27783168, 27787264) 0 0 00000080:00200000:2.0:1789760786.713879:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.713897:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6783, 6783] 00020000:00200000:2.0:1789760786.713907:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27783168 chunk: [27783168, 27787264) 27787264, 4096 00020000:00200000:2.0:1789760786.713912:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.713932:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6783, 6783] 00000008:00010000:2.0:1789760786.713950:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.713955:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.713966:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27783168, 27787264) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.713979:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27783168=0x1a7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.714118:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.714121:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27787264 00000020:00010000:2.0:1789760786.714130:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.714133:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.714137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.714140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.714143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.714151:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.714158:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.714175:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760786.714282:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 27783168<>27787264 00000100:00100000:3.0:1789760786.714345:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853952:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.755676:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974102 gen 3 00000008:00010000:3.0:1789760786.755866:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760786.755877:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.755889:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.755912:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760786.755945:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493853952:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.756624:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6783, 6783] 00000080:00200000:2.0:1789760786.756642:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.756649:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.756664:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27787264 i_blocks 55305 00000020:00200000:2.0:1789760786.756669:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.756680:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.757116:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27787264, count: 4096 00000080:00200000:2.0:1789760786.757126:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27787264, bytes: 4096 00000080:00200000:2.0:1789760786.757136:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.757144:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.757147:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27787264, 27791360) 0 0 00000080:00200000:2.0:1789760786.757151:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.757168:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6784, 6784] 00020000:00200000:2.0:1789760786.757176:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27787264 chunk: [27787264, 27791360) 27791360, 4096 00020000:00200000:2.0:1789760786.757180:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.757273:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6784, 6784] 00000008:00010000:2.0:1789760786.757293:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.757299:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.757312:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27787264, 27791360) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.757327:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27787264=0x1a80000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.757427:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.757652:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27791360 00000020:00010000:2.0:1789760786.757662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.757665:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.757669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.757672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.757675:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.757683:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.757690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.758162:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760786.758689:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 27787264<>27791360 00000100:00100000:3.0:1789760786.759150:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.759181:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.791643:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974103 gen 3 00000008:00010000:3.0:1789760786.791691:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760786.791700:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.791711:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.791878:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760786.791908:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854080:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.791986:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6784, 6784] 00000080:00200000:2.0:1789760786.792012:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.792018:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.792033:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27791360 i_blocks 55305 00000020:00200000:2.0:1789760786.792036:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.792047:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.792080:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27791360, count: 4096 00000080:00200000:2.0:1789760786.792088:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27791360, bytes: 4096 00000080:00200000:2.0:1789760786.792096:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.797333:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.797338:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27791360, 27795456) 0 0 00000080:00200000:2.0:1789760786.797346:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.797490:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6785, 6785] 00020000:00200000:2.0:1789760786.797502:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27791360 chunk: [27791360, 27795456) 27795456, 4096 00020000:00200000:2.0:1789760786.797508:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.797536:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6785, 6785] 00000008:00010000:2.0:1789760786.797563:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.797569:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.797584:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27791360, 27795456) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.797601:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27791360=0x1a81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.797672:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.797676:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27795456 00000020:00010000:2.0:1789760786.797685:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.797689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.797691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.797695:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.797698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.797709:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.797717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.797744:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760786.797860:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 27791360<>27795456 00000100:00100000:3.0:1789760786.798192:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854208:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.835860:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974104 gen 3 00000008:00010000:3.0:1789760786.835910:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760786.835918:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.836006:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.836089:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760786.836122:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854208:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.836738:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6785, 6785] 00000080:00200000:2.0:1789760786.836764:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.836772:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.836789:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27795456 i_blocks 55305 00000020:00200000:2.0:1789760786.836793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.836804:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.836839:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27795456, count: 4096 00000080:00200000:2.0:1789760786.836846:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27795456, bytes: 4096 00000080:00200000:2.0:1789760786.836857:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.836866:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.836868:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27795456, 27799552) 0 0 00000080:00200000:2.0:1789760786.836872:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.836887:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6786, 6786] 00020000:00200000:2.0:1789760786.836896:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27795456 chunk: [27795456, 27799552) 27799552, 4096 00020000:00200000:2.0:1789760786.836901:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.836922:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6786, 6786] 00000008:00010000:2.0:1789760786.837351:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.837359:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.837373:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27795456, 27799552) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.837388:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27795456=0x1a82000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.837484:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.837488:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27799552 00000020:00010000:2.0:1789760786.837497:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.837500:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.837503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.837507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.837510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.837518:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.837524:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.837544:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760786.837659:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27795456<>27799552 00000100:00100000:2.0:1789760786.837741:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493854336:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760786.898268:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760786.898285:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.898298:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.898445:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760786.898483:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493854336:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.900105:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6786, 6786] 00000080:00200000:2.0:1789760786.900129:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.900136:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.900154:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27799552 i_blocks 55305 00000020:00200000:2.0:1789760786.900160:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.900175:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.900208:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27799552, count: 4096 00000080:00200000:2.0:1789760786.900217:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27799552, bytes: 4096 00000080:00200000:2.0:1789760786.900229:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.900238:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.900240:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27799552, 27803648) 0 0 00000080:00200000:2.0:1789760786.900244:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.900260:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6787, 6787] 00020000:00200000:2.0:1789760786.900269:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27799552 chunk: [27799552, 27803648) 27803648, 4096 00020000:00200000:2.0:1789760786.900275:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.900296:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6787, 6787] 00000008:00010000:2.0:1789760786.900317:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.900325:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.900337:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27799552, 27803648) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.900352:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27799552=0x1a83000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.900431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.900434:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27803648 00000020:00010000:2.0:1789760786.900442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.900446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.900449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.900452:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.900454:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.900462:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.900469:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.900620:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760786.900808:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27799552<>27803648 00000100:00100000:3.0:1789760786.900967:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854464:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.946264:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974106 gen 3 00000008:00010000:3.0:1789760786.946329:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760786.946337:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.946348:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.946369:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760786.946434:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854464:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.946467:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6787, 6787] 00000080:00200000:2.0:1789760786.946538:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.946545:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.946561:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27803648 i_blocks 55305 00000020:00200000:2.0:1789760786.946565:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.946576:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.946612:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27803648, count: 4096 00000080:00200000:2.0:1789760786.946619:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27803648, bytes: 4096 00000080:00200000:2.0:1789760786.946628:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.946636:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.946639:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27803648, 27807744) 0 0 00000080:00200000:2.0:1789760786.946643:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.946659:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6788, 6788] 00020000:00200000:2.0:1789760786.946668:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27803648 chunk: [27803648, 27807744) 27807744, 4096 00020000:00200000:2.0:1789760786.946672:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.946693:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6788, 6788] 00000008:00010000:2.0:1789760786.946711:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.946717:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.946730:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27803648, 27807744) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.946744:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27803648=0x1a84000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.946868:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.946871:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27807744 00000020:00010000:2.0:1789760786.946880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.946883:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.946887:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.946890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.946893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.946900:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.946906:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.946922:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760786.947033:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27803648<>27807744 00000100:00100000:3.0:1789760786.947128:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.947145:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854592:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760786.982766:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974107 gen 3 00000008:00010000:3.0:1789760786.982837:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760786.982846:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.982857:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.982877:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760786.982908:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854592:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760786.983128:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6788, 6788] 00000080:00200000:2.0:1789760786.983146:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760786.983153:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760786.983168:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27807744 i_blocks 55305 00000020:00200000:2.0:1789760786.983172:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760786.983183:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760786.983222:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27807744, count: 4096 00000080:00200000:2.0:1789760786.983230:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27807744, bytes: 4096 00000080:00200000:2.0:1789760786.983241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760786.983250:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760786.983252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27807744, 27811840) 0 0 00000080:00200000:2.0:1789760786.983267:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760786.983284:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6789, 6789] 00020000:00200000:2.0:1789760786.983293:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27807744 chunk: [27807744, 27811840) 27811840, 4096 00020000:00200000:2.0:1789760786.983297:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760786.983317:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6789, 6789] 00000008:00010000:2.0:1789760786.983336:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760786.983341:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760786.983354:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27807744, 27811840) append 0 append lockpos 0 00000080:00200000:2.0:1789760786.983368:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27807744=0x1a85000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760786.983459:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760786.983462:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27811840 00000020:00010000:2.0:1789760786.983480:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760786.983485:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.983489:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760786.983492:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760786.983495:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760786.983504:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760786.983512:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760786.983529:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760786.983636:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27807744<>27811840 00000100:00100000:3.0:1789760786.983977:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854720:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.018184:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974108 gen 3 00000008:00010000:3.0:1789760787.018231:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760787.018239:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.018251:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.018350:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760787.018382:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854720:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.018668:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6789, 6789] 00000080:00200000:2.0:1789760787.018687:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.018694:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.018710:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27811840 i_blocks 55305 00000020:00200000:2.0:1789760787.018715:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.018726:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.018766:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27811840, count: 4096 00000080:00200000:2.0:1789760787.018775:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27811840, bytes: 4096 00000080:00200000:2.0:1789760787.018784:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.018793:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.018795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27811840, 27815936) 0 0 00000080:00200000:2.0:1789760787.018801:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.018818:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6790, 6790] 00020000:00200000:2.0:1789760787.018827:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27811840 chunk: [27811840, 27815936) 27815936, 4096 00020000:00200000:2.0:1789760787.018832:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.018854:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6790, 6790] 00000008:00010000:2.0:1789760787.018879:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.018888:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.018903:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27811840, 27815936) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.018920:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27811840=0x1a86000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.018981:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.018984:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27815936 00000020:00010000:2.0:1789760787.019003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.019008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.019011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.019014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.019017:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.019025:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.019031:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.019055:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760787.019157:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 27811840<>27815936 00000100:00100000:3.0:1789760787.019646:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.019673:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854848:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.061666:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974109 gen 3 00000008:00010000:3.0:1789760787.061724:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760787.061733:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.061743:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.061764:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760787.061805:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854848:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.062089:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6790, 6790] 00000080:00200000:2.0:1789760787.062106:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.062113:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.062129:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27815936 i_blocks 55305 00000020:00200000:2.0:1789760787.062133:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.062144:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.062179:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27815936, count: 4096 00000080:00200000:2.0:1789760787.062188:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27815936, bytes: 4096 00000080:00200000:2.0:1789760787.062198:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.062207:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.062209:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27815936, 27820032) 0 0 00000080:00200000:2.0:1789760787.062214:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.062229:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6791, 6791] 00020000:00200000:2.0:1789760787.062239:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27815936 chunk: [27815936, 27820032) 27820032, 4096 00020000:00200000:2.0:1789760787.062243:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.062339:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6791, 6791] 00000008:00010000:2.0:1789760787.062357:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.062363:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.062375:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27815936, 27820032) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.062389:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27815936=0x1a87000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.062520:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.062523:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27820032 00000020:00010000:2.0:1789760787.062532:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.062536:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.062539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.062543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.062546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.062553:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.062561:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.062574:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760787.062680:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 27815936<>27820032 00000100:00100000:3.0:1789760787.062752:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854976:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760787.103187:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760787.103204:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.103218:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.103237:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760787.103267:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493854976:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.103351:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6791, 6791] 00000080:00200000:2.0:1789760787.103368:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.103374:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.103389:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27820032 i_blocks 55305 00000020:00200000:2.0:1789760787.103419:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.103430:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.103465:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27820032, count: 4096 00000080:00200000:2.0:1789760787.103482:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27820032, bytes: 4096 00000080:00200000:2.0:1789760787.103492:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.103501:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.103504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27820032, 27824128) 0 0 00000080:00200000:2.0:1789760787.103509:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.103524:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6792, 6792] 00020000:00200000:2.0:1789760787.103533:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27820032 chunk: [27820032, 27824128) 27824128, 4096 00020000:00200000:2.0:1789760787.103538:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.103557:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6792, 6792] 00000008:00010000:2.0:1789760787.103575:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.103581:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.103592:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27820032, 27824128) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.103605:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27820032=0x1a88000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.103657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.103659:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27824128 00000020:00010000:2.0:1789760787.103668:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.103671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.103674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.103678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.103681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.103689:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.103696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.103711:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760787.103814:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 27820032<>27824128 00000100:00100000:3.0:1789760787.103957:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.103974:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855104:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.150286:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974111 gen 3 00000008:00010000:3.0:1789760787.150353:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760787.150362:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.150373:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.150724:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6792, 6792] 00000080:00200000:2.0:1789760787.150744:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.150750:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.150767:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27824128 i_blocks 55305 00000020:00200000:2.0:1789760787.150771:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.150793:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.150830:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27824128, count: 4096 00000080:00200000:2.0:1789760787.150839:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27824128, bytes: 4096 00000080:00200000:2.0:1789760787.150849:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.150858:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.150860:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27824128, 27828224) 0 0 00000080:00200000:2.0:1789760787.150865:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.150882:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6793, 6793] 00020000:00200000:2.0:1789760787.150891:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27824128 chunk: [27824128, 27828224) 27828224, 4096 00020000:00200000:2.0:1789760787.150895:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.150914:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6793, 6793] 00000008:00010000:2.0:1789760787.150932:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.150939:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.150950:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27824128, 27828224) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.150964:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27824128=0x1a89000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.151035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.151038:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27828224 00000008:00010000:3.0:1789760787.151561:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760787.151591:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855104:192.168.203.160@tcp:4:dd.0 00000020:00010000:2.0:1789760787.151760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.151765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.151768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.151772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.151775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.151783:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.151791:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.151814:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760787.152088:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27824128<>27828224 00000100:00100000:3.0:1789760787.152199:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855232:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.200252:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974112 gen 3 00000008:00010000:2.0:1789760787.200640:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:2.0:1789760787.200650:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.200664:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.200731:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:2.0:1789760787.200762:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855232:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760787.201298:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6793, 6793] 00000080:00200000:3.0:1789760787.201316:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.201324:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.201444:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27828224 i_blocks 55305 00000020:00200000:2.0:1789760787.201449:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.201461:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.201569:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27828224, count: 4096 00000080:00200000:2.0:1789760787.201579:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27828224, bytes: 4096 00000080:00200000:2.0:1789760787.201593:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.201602:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.201605:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27828224, 27832320) 0 0 00000080:00200000:2.0:1789760787.201610:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.201626:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6794, 6794] 00020000:00200000:2.0:1789760787.201634:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27828224 chunk: [27828224, 27832320) 27832320, 4096 00020000:00200000:2.0:1789760787.201639:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.201660:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6794, 6794] 00000008:00010000:2.0:1789760787.201678:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.201684:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.201696:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27828224, 27832320) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.201711:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27828224=0x1a8a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.201767:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.201770:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27832320 00000020:00010000:2.0:1789760787.201778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.201782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.201785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.201789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.201792:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.201799:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.201807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.201850:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760787.201950:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27828224<>27832320 00000100:00100000:3.0:1789760787.202033:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855360:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760787.239611:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760787.239626:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.239640:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.239661:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760787.239691:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.239724:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6794, 6794] 00000080:00200000:2.0:1789760787.239740:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.239746:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.239762:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27832320 i_blocks 55305 00000020:00200000:2.0:1789760787.239766:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.239777:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.239813:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27832320, count: 4096 00000080:00200000:2.0:1789760787.239821:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27832320, bytes: 4096 00000080:00200000:2.0:1789760787.239830:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.239839:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.239841:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27832320, 27836416) 0 0 00000080:00200000:2.0:1789760787.239845:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.239863:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6795, 6795] 00020000:00200000:2.0:1789760787.239872:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27832320 chunk: [27832320, 27836416) 27836416, 4096 00020000:00200000:2.0:1789760787.239876:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.239896:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6795, 6795] 00000008:00010000:2.0:1789760787.239914:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.239920:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.239932:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27832320, 27836416) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.239946:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27832320=0x1a8b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.240017:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.240020:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27836416 00000020:00010000:2.0:1789760787.240028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.240032:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.240036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.240039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.240042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.240050:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.240056:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.240072:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760787.240169:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27832320<>27836416 00000100:00100000:3.0:1789760787.240228:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.240245:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855488:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.286603:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974113 gen 3 00000100:00100000:3.0:1789760787.286811:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80e00 x1876699493855488/t4294974114(4294974114) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.286864:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760787.286872:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.286883:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.286947:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760787.286970:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855488:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.287247:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6795, 6795] 00000080:00200000:2.0:1789760787.287265:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.287271:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.287287:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27836416 i_blocks 55305 00000020:00200000:2.0:1789760787.287291:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.287304:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.287340:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27836416, count: 4096 00000080:00200000:2.0:1789760787.287347:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27836416, bytes: 4096 00000080:00200000:2.0:1789760787.287358:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.287367:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.287370:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27836416, 27840512) 0 0 00000080:00200000:2.0:1789760787.287374:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.287392:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6796, 6796] 00020000:00200000:2.0:1789760787.287429:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27836416 chunk: [27836416, 27840512) 27840512, 4096 00020000:00200000:2.0:1789760787.287434:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.287455:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6796, 6796] 00000008:00010000:2.0:1789760787.287592:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.287599:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.287611:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27836416, 27840512) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.287625:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27836416=0x1a8c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.287807:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.287810:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27840512 00000020:00010000:2.0:1789760787.287819:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.287823:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.287826:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.287829:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.287832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.287839:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.287846:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.287862:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760787.287966:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27836416<>27840512 00000100:00100000:3.0:1789760787.288113:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855616:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.345651:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974114 gen 3 00000100:00100000:3.0:1789760787.345696:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83480 x1876699493855616/t4294974115(4294974115) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.345750:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760787.345757:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.345768:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.345789:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760787.345813:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493855616:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.345899:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6796, 6796] 00000080:00200000:2.0:1789760787.345914:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.345919:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.345935:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27840512 i_blocks 55305 00000020:00200000:2.0:1789760787.345938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.345948:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.345981:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27840512, count: 4096 00000080:00200000:2.0:1789760787.345998:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27840512, bytes: 4096 00000080:00200000:2.0:1789760787.346008:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.346017:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.346019:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27840512, 27844608) 0 0 00000080:00200000:2.0:1789760787.346023:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.346038:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6797, 6797] 00020000:00200000:2.0:1789760787.346047:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27840512 chunk: [27840512, 27844608) 27844608, 4096 00020000:00200000:2.0:1789760787.346052:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.346071:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6797, 6797] 00000008:00010000:2.0:1789760787.346088:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.346094:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.346106:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27840512, 27844608) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.346121:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27840512=0x1a8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.346177:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.346179:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27844608 00000020:00010000:2.0:1789760787.346187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.346191:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.346193:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.346197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.346200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.346208:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.346215:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.346229:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760787.346328:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27840512<>27844608 00000100:00100000:3.0:1789760787.346428:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855744:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.383219:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974116 gen 3 00000008:00010000:3.0:1789760787.383288:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760787.383296:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.383307:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.383322:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760787.383349:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855744:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.383509:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6797, 6797] 00000080:00200000:2.0:1789760787.383524:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.383530:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.383545:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27844608 i_blocks 55305 00000020:00200000:2.0:1789760787.383548:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.383559:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.383590:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27844608, count: 4096 00000080:00200000:2.0:1789760787.383596:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27844608, bytes: 4096 00000080:00200000:2.0:1789760787.383605:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.383613:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.383615:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27844608, 27848704) 0 0 00000080:00200000:2.0:1789760787.383619:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.383634:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6798, 6798] 00020000:00200000:2.0:1789760787.383642:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27844608 chunk: [27844608, 27848704) 27848704, 4096 00020000:00200000:2.0:1789760787.383646:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.383665:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6798, 6798] 00000008:00010000:2.0:1789760787.383680:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.383686:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.383697:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27844608, 27848704) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.383709:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27844608=0x1a8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.383758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.383761:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27848704 00000020:00010000:2.0:1789760787.383768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.383772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.383775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.383778:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.383781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.383788:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.383793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.383806:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760787.383909:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27844608<>27848704 00000100:00100000:3.0:1789760787.384157:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760787.384177:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855872:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.422666:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974117 gen 3 00000008:00010000:3.0:1789760787.422716:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760787.422724:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.422736:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.422756:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760787.422785:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493855872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.422941:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6798, 6798] 00000080:00200000:2.0:1789760787.422958:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.422964:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.422981:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27848704 i_blocks 55305 00000020:00200000:2.0:1789760787.422985:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.423007:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.423040:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27848704, count: 4096 00000080:00200000:2.0:1789760787.423057:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27848704, bytes: 4096 00000080:00200000:2.0:1789760787.423067:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.423084:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.423087:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27848704, 27852800) 0 0 00000080:00200000:2.0:1789760787.423091:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.423108:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6799, 6799] 00020000:00200000:2.0:1789760787.423117:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27848704 chunk: [27848704, 27852800) 27852800, 4096 00020000:00200000:2.0:1789760787.423121:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.423143:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6799, 6799] 00000008:00010000:2.0:1789760787.423163:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.423169:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.423181:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27848704, 27852800) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.423194:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27848704=0x1a8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.423248:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.423251:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27852800 00000020:00010000:2.0:1789760787.423259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.423263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.423266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.423270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.423273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.423280:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.423286:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.423300:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760787.423779:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27848704<>27852800 00000100:00100000:3.0:1789760787.423845:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856000:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760787.463816:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760787.463832:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.463845:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.463867:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760787.463898:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.464005:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6799, 6799] 00000080:00200000:2.0:1789760787.464020:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.464026:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.464049:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27852800 i_blocks 55305 00000020:00200000:2.0:1789760787.464054:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.464065:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.464100:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27852800, count: 4096 00000080:00200000:2.0:1789760787.464107:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27852800, bytes: 4096 00000080:00200000:2.0:1789760787.464116:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.464125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.464128:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27852800, 27856896) 0 0 00000080:00200000:2.0:1789760787.464133:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.464148:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6800, 6800] 00020000:00200000:2.0:1789760787.464157:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27852800 chunk: [27852800, 27856896) 27856896, 4096 00020000:00200000:2.0:1789760787.464161:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.464183:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6800, 6800] 00000008:00010000:2.0:1789760787.464203:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.464211:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.464223:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27852800, 27856896) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.464239:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27852800=0x1a90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.464299:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.464302:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27856896 00000020:00010000:2.0:1789760787.464311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.464315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.464319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.464323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.464326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.464334:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.464343:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.464360:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760787.464634:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27852800<>27856896 00000100:00100000:3.0:1789760787.465665:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.511094:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974118 gen 3 00000100:00100000:3.0:1789760787.511137:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81f80 x1876699493856128/t4294974119(4294974119) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.511194:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760787.511202:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.511214:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.511237:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760787.511263:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856128:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.511348:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6800, 6800] 00000080:00200000:2.0:1789760787.511365:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.511371:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.511390:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27856896 i_blocks 55305 00000020:00200000:2.0:1789760787.511423:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.511434:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.511477:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27856896, count: 4096 00000080:00200000:2.0:1789760787.511484:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27856896, bytes: 4096 00000080:00200000:2.0:1789760787.511494:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.511502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.511504:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27856896, 27860992) 0 0 00000080:00200000:2.0:1789760787.511509:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.511527:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6801, 6801] 00020000:00200000:2.0:1789760787.511536:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27856896 chunk: [27856896, 27860992) 27860992, 4096 00020000:00200000:2.0:1789760787.511541:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.511562:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6801, 6801] 00000008:00010000:2.0:1789760787.511580:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.511587:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.511599:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27856896, 27860992) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.511613:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27856896=0x1a91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.511671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.511674:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27860992 00000020:00010000:2.0:1789760787.511697:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.511701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.511704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.511707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.511710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.511718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.511725:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.511741:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760787.511838:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 27856896<>27860992 00000100:00100000:3.0:1789760787.511997:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856256:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.545949:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974119 gen 3 00000100:00100000:3.0:1789760787.546003:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80380 x1876699493856256/t4294974120(4294974120) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.546062:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760787.546070:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.546080:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.546097:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760787.546122:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856256:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.546259:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6801, 6801] 00000080:00200000:2.0:1789760787.546276:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.546282:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.546297:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27860992 i_blocks 55305 00000020:00200000:2.0:1789760787.546301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.546312:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.546345:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27860992, count: 4096 00000080:00200000:2.0:1789760787.546352:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27860992, bytes: 4096 00000080:00200000:2.0:1789760787.546360:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.546368:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.546370:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27860992, 27865088) 0 0 00000080:00200000:2.0:1789760787.546375:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.546390:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6802, 6802] 00020000:00200000:2.0:1789760787.546427:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27860992 chunk: [27860992, 27865088) 27865088, 4096 00020000:00200000:2.0:1789760787.546432:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.546453:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6802, 6802] 00000008:00010000:2.0:1789760787.546481:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.546488:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.546501:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27860992, 27865088) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.546515:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27860992=0x1a92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.546575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.546578:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27865088 00000020:00010000:2.0:1789760787.546586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.546590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.546593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.546597:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.546599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.546606:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.546613:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.546627:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760787.546728:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 27860992<>27865088 00000100:00100000:3.0:1789760787.546890:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.581080:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974120 gen 3 00000100:00100000:3.0:1789760787.581124:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80000 x1876699493856384/t4294974121(4294974121) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.581174:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760787.581182:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.581193:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.581214:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760787.581238:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856384:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.581331:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6802, 6802] 00000080:00200000:2.0:1789760787.581349:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.581355:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.581370:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27865088 i_blocks 55305 00000020:00200000:2.0:1789760787.581373:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.581383:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.581445:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27865088, count: 4096 00000080:00200000:2.0:1789760787.581453:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27865088, bytes: 4096 00000080:00200000:2.0:1789760787.581462:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.581484:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.581486:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27865088, 27869184) 0 0 00000080:00200000:2.0:1789760787.581491:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.581508:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6803, 6803] 00020000:00200000:2.0:1789760787.581517:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27865088 chunk: [27865088, 27869184) 27869184, 4096 00020000:00200000:2.0:1789760787.581523:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.581544:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6803, 6803] 00000008:00010000:2.0:1789760787.581561:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.581566:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.581578:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27865088, 27869184) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.581591:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27865088=0x1a93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.581648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.581650:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27869184 00000020:00010000:2.0:1789760787.581659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.581663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.581666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.581670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.581673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.581680:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.581686:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.581700:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760787.581798:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27865088<>27869184 00000100:00100000:3.0:1789760787.581957:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856512:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.619760:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974121 gen 3 00000100:00100000:3.0:1789760787.619803:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80a80 x1876699493856512/t4294974122(4294974122) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760787.619851:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760787.619858:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.619868:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.619937:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760787.619960:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856512:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.620014:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6803, 6803] 00000080:00200000:2.0:1789760787.620032:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.620038:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.620054:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27869184 i_blocks 55305 00000020:00200000:2.0:1789760787.620058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.620068:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.620104:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27869184, count: 4096 00000080:00200000:2.0:1789760787.620113:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27869184, bytes: 4096 00000080:00200000:2.0:1789760787.620123:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.620132:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.620134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27869184, 27873280) 0 0 00000080:00200000:2.0:1789760787.620139:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.620155:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6804, 6804] 00020000:00200000:2.0:1789760787.620163:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27869184 chunk: [27869184, 27873280) 27873280, 4096 00020000:00200000:2.0:1789760787.620167:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.620188:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6804, 6804] 00000008:00010000:2.0:1789760787.620205:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.620211:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.620222:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27869184, 27873280) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.620236:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27869184=0x1a94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.620290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.620293:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27873280 00000020:00010000:2.0:1789760787.620302:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.620306:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.620309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.620312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.620315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.620323:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.620330:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.620346:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760787.620490:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27869184<>27873280 00000100:00100000:3.0:1789760787.620631:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856640:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.651350:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974123 gen 3 00000008:00010000:3.0:1789760787.651447:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760787.651456:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.651466:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.651488:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760787.651518:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493856640:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760787.651667:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6804, 6804] 00000080:00200000:2.0:1789760787.651684:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.651690:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.651707:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27873280 i_blocks 55305 00000020:00200000:2.0:1789760787.651712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.651722:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.651758:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27873280, count: 4096 00000080:00200000:2.0:1789760787.651765:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27873280, bytes: 4096 00000080:00200000:2.0:1789760787.651775:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.651783:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.651785:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27873280, 27877376) 0 0 00000080:00200000:2.0:1789760787.651790:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.651807:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6805, 6805] 00020000:00200000:2.0:1789760787.651816:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27873280 chunk: [27873280, 27877376) 27877376, 4096 00020000:00200000:2.0:1789760787.651821:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.651841:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6805, 6805] 00000008:00010000:2.0:1789760787.651861:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.651867:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760787.651879:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27873280, 27877376) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.651894:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27873280=0x1a95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.651957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.651960:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27877376 00000020:00010000:2.0:1789760787.651969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.651973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.651976:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.651980:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.651983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.652000:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.652008:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.652029:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760787.652128:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 27873280<>27877376 00000100:00100000:3.0:1789760787.652194:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856768:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760787.688381:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760787.688426:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.688437:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.688457:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760787.688493:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493856768:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760787.688630:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6805, 6805] 00000080:00200000:0.0:1789760787.688646:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.688652:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.689253:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27877376 i_blocks 55305 00000020:00200000:0.0:1789760787.689262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.689276:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.689388:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27877376, count: 4096 00000080:00200000:0.0:1789760787.689430:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27877376, bytes: 4096 00000080:00200000:0.0:1789760787.689443:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.689453:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.689456:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27877376, 27881472) 0 0 00000080:00200000:0.0:1789760787.689461:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.689479:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6806, 6806] 00020000:00200000:0.0:1789760787.689488:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27877376 chunk: [27877376, 27881472) 27881472, 4096 00020000:00200000:0.0:1789760787.689492:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.689513:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6806, 6806] 00000008:00010000:0.0:1789760787.689531:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.689539:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760787.689551:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27877376, 27881472) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.689566:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27877376=0x1a96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.689631:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.689634:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27881472 00000020:00010000:0.0:1789760787.689643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.689646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.689649:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.689653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac038 (ffff8eb6450b4248/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.689724:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4248/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.689735:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.689743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.689769:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760787.689890:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 27877376<>27881472 00000100:00100000:1.0:1789760787.690008:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493856896:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760787.696724:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760787.733254:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974125 gen 3 00000008:00010000:1.0:1789760787.733553:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760787.733563:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.733575:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.733599:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760787.733632:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493856896:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760787.733904:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6806, 6806] 00000080:00200000:3.0:1789760787.733922:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.733928:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.734285:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27881472 i_blocks 55305 00000020:00200000:3.0:1789760787.734290:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.734302:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.734511:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27881472, count: 4096 00000080:00200000:3.0:1789760787.734520:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27881472, bytes: 4096 00000080:00200000:3.0:1789760787.734535:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.734544:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.734546:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27881472, 27885568) 0 0 00000080:00200000:3.0:1789760787.734551:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.734568:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6807, 6807] 00020000:00200000:3.0:1789760787.734578:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27881472 chunk: [27881472, 27885568) 27885568, 4096 00020000:00200000:3.0:1789760787.734583:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.734603:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6807, 6807] 00000008:00010000:3.0:1789760787.734621:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.734628:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760787.734640:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27881472, 27885568) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.734654:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27881472=0x1a97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.734719:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.734722:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27885568 00000020:00010000:3.0:1789760787.734731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.734734:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.734738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.734741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.734743:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.734751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.734759:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.734784:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760787.734889:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27881472<>27885568 00000100:00100000:2.0:1789760787.734980:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493857024:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760787.807349:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974126 gen 3 00000008:00010000:3.0:1789760787.807429:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760787.807439:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.807450:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.807478:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760787.807506:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493857024:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760787.807624:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6807, 6807] 00000080:00200000:0.0:1789760787.807641:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.807773:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.807885:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27885568 i_blocks 55305 00000020:00200000:0.0:1789760787.807890:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.807900:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.807933:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27885568, count: 4096 00000080:00200000:0.0:1789760787.807947:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27885568, bytes: 4096 00000080:00200000:0.0:1789760787.807957:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.808271:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.808274:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27885568, 27889664) 0 0 00000080:00200000:0.0:1789760787.808280:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.808297:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6808, 6808] 00020000:00200000:0.0:1789760787.808307:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27885568 chunk: [27885568, 27889664) 27889664, 4096 00020000:00200000:0.0:1789760787.808312:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.808333:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6808, 6808] 00000008:00010000:0.0:1789760787.808365:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.808372:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760787.808384:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27885568, 27889664) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.808429:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27885568=0x1a98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.808491:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.808495:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27889664 00000020:00010000:0.0:1789760787.808503:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.808507:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.808510:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.808526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.808529:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.808538:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.808545:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.808564:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760787.808670:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 27885568<>27889664 00000100:00100000:1.0:1789760787.809044:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.809069:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493857152:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760787.882706:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974127 gen 3 00000008:00010000:1.0:1789760787.882761:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760787.882770:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.882871:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.882901:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760787.882933:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493857152:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760787.885832:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6808, 6808] 00000080:00200000:0.0:1789760787.885858:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.885865:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.885883:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27889664 i_blocks 55305 00000020:00200000:0.0:1789760787.885888:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.885902:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.885933:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27889664, count: 4096 00000080:00200000:0.0:1789760787.885941:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27889664, bytes: 4096 00000080:00200000:0.0:1789760787.885956:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.885965:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.885967:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27889664, 27893760) 0 0 00000080:00200000:0.0:1789760787.885971:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.886000:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6809, 6809] 00020000:00200000:0.0:1789760787.886008:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27889664 chunk: [27889664, 27893760) 27893760, 4096 00020000:00200000:0.0:1789760787.886014:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.886035:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6809, 6809] 00000008:00010000:0.0:1789760787.886053:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.886058:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760787.886069:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27889664, 27893760) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.886082:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27889664=0x1a99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.886145:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.886148:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27893760 00000020:00010000:0.0:1789760787.886157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.886160:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.886164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.886169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.886173:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.886180:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.886188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.886205:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760787.886315:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 27889664<>27893760 00000100:00100000:0.0:1789760787.886370:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760787.886428:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493857280:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760787.962224:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760787.962242:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.962254:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.962277:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000080:00200000:3.0:1789760787.962307:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6809, 6809] 00000100:00100000:0.0:1789760787.962308:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493857280:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760787.962325:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.962330:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.962345:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27893760 i_blocks 55305 00000020:00200000:3.0:1789760787.962349:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.962359:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.962423:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27893760, count: 4096 00000080:00200000:3.0:1789760787.962431:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27893760, bytes: 4096 00000080:00200000:3.0:1789760787.962442:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.962451:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.962453:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27893760, 27897856) 0 0 00000080:00200000:3.0:1789760787.962458:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.962545:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6810, 6810] 00020000:00200000:3.0:1789760787.962555:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27893760 chunk: [27893760, 27897856) 27897856, 4096 00020000:00200000:3.0:1789760787.962559:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.962581:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6810, 6810] 00000008:00010000:3.0:1789760787.962601:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.962607:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760787.962620:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27893760, 27897856) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.962635:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27893760=0x1a9a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.962691:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.962694:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27897856 00000020:00010000:3.0:1789760787.962701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.962704:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.962707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.962710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.962713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.962720:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.962726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.962741:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760787.963621:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 27893760<>27897856 00000100:00100000:3.0:1789760787.963679:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.963700:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493857408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.013362:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493857536:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760788.013947:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493857664:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760788.014160:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760788.014168:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493857792:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760788.020359:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493857792:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760788.020379:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493857664:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760788.020903:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493857536:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760788.026741:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974128 gen 3 00000100:00100000:3.0:1789760788.026788:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149880 x1876699493857408/t4294974129(4294974129) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760788.026840:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760788.026847:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.026858:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.026879:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760788.026902:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493857408:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.027000:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6810, 6810] 00000080:00200000:0.0:1789760788.027017:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.027022:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.027037:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27897856 i_blocks 55305 00000020:00200000:0.0:1789760788.027042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.027052:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.027084:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27897856, count: 4096 00000080:00200000:0.0:1789760788.027091:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27897856, bytes: 4096 00000080:00200000:0.0:1789760788.027101:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.027110:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.027112:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27897856, 27901952) 0 0 00000080:00200000:0.0:1789760788.027116:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.027133:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6811, 6811] 00020000:00200000:0.0:1789760788.027142:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27897856 chunk: [27897856, 27901952) 27901952, 4096 00020000:00200000:0.0:1789760788.027146:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.027174:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6811, 6811] 00000008:00010000:0.0:1789760788.027192:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.027199:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760788.027211:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27897856, 27901952) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.027225:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27897856=0x1a9b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.027282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.027285:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27901952 00000020:00010000:0.0:1789760788.027294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.027297:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.027301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.027304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.027307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.027316:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.027323:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.027337:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760788.027487:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 27897856<>27901952 00000100:00100000:1.0:1789760788.027584:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.027604:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493857920:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760788.084461:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974130 gen 3 00000008:00010000:1.0:1789760788.084527:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760788.084536:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.084545:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.085354:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760788.085392:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493857920:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.098298:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6811, 6811] 00000080:00200000:0.0:1789760788.098330:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.098337:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.098380:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27901952 i_blocks 55305 00000020:00200000:0.0:1789760788.098385:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.098429:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.098463:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27901952, count: 4096 00000080:00200000:0.0:1789760788.098473:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27901952, bytes: 4096 00000080:00200000:0.0:1789760788.098488:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.098498:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.098500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27901952, 27906048) 0 0 00000080:00200000:0.0:1789760788.098504:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.098520:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6812, 6812] 00020000:00200000:0.0:1789760788.098529:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27901952 chunk: [27901952, 27906048) 27906048, 4096 00020000:00200000:0.0:1789760788.098532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.098555:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6812, 6812] 00000008:00010000:0.0:1789760788.098574:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.098581:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760788.098592:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27901952, 27906048) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.098608:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27901952=0x1a9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.098672:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.098675:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27906048 00000020:00010000:0.0:1789760788.098683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.098687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.098690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.098693:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.098696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.098705:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.098712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.098730:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760788.098839:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 27901952<>27906048 00000100:00100000:0.0:1789760788.098884:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760788.098909:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493858048:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760788.149304:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974131 gen 3 00000008:00010000:1.0:1789760788.149357:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760788.149365:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.149375:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.149432:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760788.149460:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493858048:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.149707:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6812, 6812] 00000080:00200000:0.0:1789760788.149722:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.149729:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.149744:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27906048 i_blocks 55305 00000020:00200000:0.0:1789760788.149748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.149757:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.149790:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27906048, count: 4096 00000080:00200000:0.0:1789760788.149798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27906048, bytes: 4096 00000080:00200000:0.0:1789760788.149807:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.149815:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.149817:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27906048, 27910144) 0 0 00000080:00200000:0.0:1789760788.149822:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.149837:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6813, 6813] 00020000:00200000:0.0:1789760788.149845:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27906048 chunk: [27906048, 27910144) 27910144, 4096 00020000:00200000:0.0:1789760788.149850:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.149865:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6813, 6813] 00000008:00010000:0.0:1789760788.149882:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.149888:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760788.149898:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27906048, 27910144) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.149911:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27906048=0x1a9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.149965:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.149967:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27910144 00000020:00010000:0.0:1789760788.149975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.149978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.149993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.149997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.150000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.150007:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.150013:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.150028:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760788.150245:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 27906048<>27910144 00000100:00100000:1.0:1789760788.150376:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.150489:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858176:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760788.201032:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974132 gen 3 00000008:00010000:1.0:1789760788.201083:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760788.201093:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.201105:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.201131:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760788.201161:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858176:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.201273:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6813, 6813] 00000080:00200000:0.0:1789760788.201291:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.201298:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.201312:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27910144 i_blocks 55305 00000020:00200000:0.0:1789760788.201316:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.201326:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.201362:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27910144, count: 4096 00000080:00200000:0.0:1789760788.201369:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27910144, bytes: 4096 00000080:00200000:0.0:1789760788.201585:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.201595:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.201597:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27910144, 27914240) 0 0 00000080:00200000:0.0:1789760788.201602:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.205289:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6814, 6814] 00020000:00200000:1.0:1789760788.205315:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27910144 chunk: [27910144, 27914240) 27914240, 4096 00020000:00200000:1.0:1789760788.205321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.205352:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6814, 6814] 00000008:00010000:1.0:1789760788.205517:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.205525:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760788.205542:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27910144, 27914240) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.205561:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27910144=0x1a9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.205636:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.205640:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27914240 00000020:00010000:1.0:1789760788.205648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.205652:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.205655:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.205658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.205660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.205675:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.205682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.205738:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760788.205853:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 27910144<>27914240 00000100:00100000:0.0:1789760788.205976:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858304:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760788.272934:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974133 gen 3 00000008:00010000:0.0:1789760788.272998:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760788.273007:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.273019:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.273045:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760788.273074:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858304:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760788.273202:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6814, 6814] 00000080:00200000:1.0:1789760788.273219:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.273225:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.273240:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27914240 i_blocks 55305 00000020:00200000:1.0:1789760788.273244:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.273254:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.273288:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27914240, count: 4096 00000080:00200000:1.0:1789760788.273295:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27914240, bytes: 4096 00000080:00200000:1.0:1789760788.273305:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.273314:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.273316:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27914240, 27918336) 0 0 00000080:00200000:1.0:1789760788.273321:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.273336:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6815, 6815] 00020000:00200000:1.0:1789760788.273344:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27914240 chunk: [27914240, 27918336) 27918336, 4096 00020000:00200000:1.0:1789760788.273349:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.273370:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6815, 6815] 00000008:00010000:1.0:1789760788.273388:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.273424:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455efe30, osc lock ffff8eb646c6aab0, flags 0x0 00000080:00200000:1.0:1789760788.273437:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27914240, 27918336) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.273454:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27914240=0x1a9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.273810:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.273814:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27918336 00000020:00010000:2.0:1789760788.273823:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.273827:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.273830:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455efe30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.273834:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.273838:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455efe30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.273849:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.273857:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.273873:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760788.273977:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 27914240<>27918336 00000100:00100000:3.0:1789760788.274157:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493858432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.320743:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974134 gen 3 00000008:00010000:3.0:1789760788.320792:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760788.320800:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.320813:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.320841:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760788.320870:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493858432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.321007:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6815, 6815] 00000080:00200000:2.0:1789760788.321022:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.321042:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.321057:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27918336 i_blocks 55305 00000020:00200000:2.0:1789760788.321061:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.321071:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.321101:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27918336, count: 4096 00000080:00200000:2.0:1789760788.321108:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27918336, bytes: 4096 00000080:00200000:2.0:1789760788.321117:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.321125:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.321127:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27918336, 27922432) 0 0 00000080:00200000:2.0:1789760788.321132:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.321147:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6816, 6816] 00020000:00200000:2.0:1789760788.321155:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27918336 chunk: [27918336, 27922432) 27922432, 4096 00020000:00200000:2.0:1789760788.321159:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.321177:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6816, 6816] 00000008:00010000:2.0:1789760788.321195:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.321200:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.321212:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27918336, 27922432) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.321225:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27918336=0x1aa0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.321283:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.321286:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27922432 00000020:00010000:2.0:1789760788.321294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.321298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.321301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.321304:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.321307:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.321314:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.321320:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.321335:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760788.322943:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 27918336<>27922432 00000100:00100000:3.0:1789760788.323028:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493858560:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760788.368227:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760788.368245:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.368257:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.368313:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760788.368339:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493858560:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.368713:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6816, 6816] 00000080:00200000:0.0:1789760788.368731:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.368737:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.368752:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27922432 i_blocks 55305 00000020:00200000:0.0:1789760788.369445:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.369461:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.369496:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27922432, count: 4096 00000080:00200000:0.0:1789760788.369504:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27922432, bytes: 4096 00000080:00200000:0.0:1789760788.369520:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.369530:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.369532:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27922432, 27926528) 0 0 00000080:00200000:0.0:1789760788.369537:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.369555:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6817, 6817] 00020000:00200000:0.0:1789760788.369570:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27922432 chunk: [27922432, 27926528) 27926528, 4096 00020000:00200000:0.0:1789760788.369575:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.369596:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6817, 6817] 00000008:00010000:0.0:1789760788.369713:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.369720:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760788.369733:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27922432, 27926528) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.369750:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27922432=0x1aa1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.369814:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.369884:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27926528 00000020:00010000:0.0:1789760788.369893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.369896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.369900:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.369903:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.369906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.369915:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.369922:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.369944:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760788.370071:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 27922432<>27926528 00000100:00100000:1.0:1789760788.370159:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493858688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760788.408824:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974136 gen 3 00000008:00010000:1.0:1789760788.408890:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760788.408898:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.408908:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.408932:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760788.408961:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493858688:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760788.409079:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6817, 6817] 00000080:00200000:0.0:1789760788.409096:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.409113:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.409130:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27926528 i_blocks 55305 00000020:00200000:0.0:1789760788.409133:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.409145:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.409180:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27926528, count: 4096 00000080:00200000:0.0:1789760788.409188:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27926528, bytes: 4096 00000080:00200000:0.0:1789760788.409199:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.409210:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.409212:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27926528, 27930624) 0 0 00000080:00200000:0.0:1789760788.409216:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.409234:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6818, 6818] 00020000:00200000:0.0:1789760788.409244:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27926528 chunk: [27926528, 27930624) 27930624, 4096 00020000:00200000:0.0:1789760788.409249:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.409269:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6818, 6818] 00000008:00010000:0.0:1789760788.409288:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.409294:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760788.409307:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27926528, 27930624) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.409324:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27926528=0x1aa2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.409381:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.409384:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27930624 00000020:00010000:0.0:1789760788.409429:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.409433:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.409437:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.409441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.409444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.409453:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.409470:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.409488:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760788.409761:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 27926528<>27930624 00000100:00100000:1.0:1789760788.410077:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858816:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760788.476513:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974137 gen 3 00000008:00010000:0.0:1789760788.476577:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760788.476586:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.476597:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.476635:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760788.476668:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493858816:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760788.476841:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6818, 6818] 00000080:00200000:3.0:1789760788.476858:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.476865:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.476881:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27930624 i_blocks 55305 00000020:00200000:3.0:1789760788.476886:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.476897:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.476929:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27930624, count: 4096 00000080:00200000:3.0:1789760788.476938:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27930624, bytes: 4096 00000080:00200000:3.0:1789760788.476951:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.476959:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.476961:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27930624, 27934720) 0 0 00000080:00200000:3.0:1789760788.476976:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.477003:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6819, 6819] 00020000:00200000:3.0:1789760788.477012:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27930624 chunk: [27930624, 27934720) 27934720, 4096 00020000:00200000:3.0:1789760788.477016:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.477037:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6819, 6819] 00000008:00010000:3.0:1789760788.477057:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.477064:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760788.477076:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27930624, 27934720) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.477091:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27930624=0x1aa3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.477161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.477164:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27934720 00000020:00010000:3.0:1789760788.477172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.477187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.477190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.477194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.477198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.477206:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.477213:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.477229:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760788.477732:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 27930624<>27934720 00000100:00100000:3.0:1789760788.477776:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.477798:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493858944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.532406:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974138 gen 3 00000008:00010000:3.0:1789760788.532462:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760788.532470:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.532494:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.532521:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000080:00200000:2.0:1789760788.532660:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6819, 6819] 00000080:00200000:2.0:1789760788.532684:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.532690:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.532709:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27934720 i_blocks 55305 00000020:00200000:2.0:1789760788.532713:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.532725:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.532763:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27934720, count: 4096 00000080:00200000:2.0:1789760788.532779:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27934720, bytes: 4096 00000080:00200000:2.0:1789760788.532795:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.532805:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.532808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27934720, 27938816) 0 0 00000080:00200000:2.0:1789760788.532812:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.532830:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6820, 6820] 00020000:00200000:2.0:1789760788.532840:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27934720 chunk: [27934720, 27938816) 27938816, 4096 00020000:00200000:2.0:1789760788.532845:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.532866:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6820, 6820] 00000008:00010000:2.0:1789760788.532885:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.532892:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.532905:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27934720, 27938816) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.532920:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27934720=0x1aa4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.532997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.532999:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27938816 00000020:00010000:2.0:1789760788.533008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.533012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.533014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.533017:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.533021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.533029:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.533035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.533055:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760788.533183:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 27934720<>27938816 00000100:00100000:2.0:1789760788.533225:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760788.533250:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859072:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.533851:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493858944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.586151:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974139 gen 3 00000008:00010000:3.0:1789760788.586376:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760788.586386:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.586433:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.586468:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760788.586498:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.586629:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6820, 6820] 00000080:00200000:2.0:1789760788.586646:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.586652:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.586667:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27938816 i_blocks 55305 00000020:00200000:2.0:1789760788.586671:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.586681:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.586715:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27938816, count: 4096 00000080:00200000:2.0:1789760788.586723:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27938816, bytes: 4096 00000080:00200000:2.0:1789760788.586734:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.586743:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.586745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27938816, 27942912) 0 0 00000080:00200000:2.0:1789760788.586750:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.586766:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6821, 6821] 00020000:00200000:2.0:1789760788.586773:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27938816 chunk: [27938816, 27942912) 27942912, 4096 00020000:00200000:2.0:1789760788.586778:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.586796:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6821, 6821] 00000008:00010000:2.0:1789760788.586816:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.586822:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.586833:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27938816, 27942912) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.586846:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27938816=0x1aa5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.586904:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.586907:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27942912 00000020:00010000:2.0:1789760788.586915:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.586918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.586922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.586926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.586929:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.586936:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.586945:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.586960:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760788.587130:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 27938816<>27942912 00000100:00100000:3.0:1789760788.587339:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.646303:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974140 gen 3 00000008:00010000:3.0:1789760788.646354:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760788.646362:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.646374:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.646478:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760788.646507:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.647968:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6821, 6821] 00000080:00200000:2.0:1789760788.648003:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.648010:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.648026:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27942912 i_blocks 55305 00000020:00200000:2.0:1789760788.648031:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.648112:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.648148:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27942912, count: 4096 00000080:00200000:2.0:1789760788.648156:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27942912, bytes: 4096 00000080:00200000:2.0:1789760788.648171:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.648282:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.648285:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27942912, 27947008) 0 0 00000080:00200000:2.0:1789760788.648290:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.648308:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6822, 6822] 00020000:00200000:2.0:1789760788.648317:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27942912 chunk: [27942912, 27947008) 27947008, 4096 00020000:00200000:2.0:1789760788.648321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.648343:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6822, 6822] 00000008:00010000:2.0:1789760788.648362:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.648368:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.648379:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27942912, 27947008) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.648416:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27942912=0x1aa6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.648540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.648543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27947008 00000020:00010000:2.0:1789760788.648552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.648555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.648560:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.648563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.648566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.648575:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.648582:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.648606:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760788.648707:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 27942912<>27947008 00000100:00100000:3.0:1789760788.648853:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859328:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760788.716805:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760788.719799:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760788.719814:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.719839:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.719857:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760788.719887:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.720138:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6822, 6822] 00000080:00200000:2.0:1789760788.720154:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.720160:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.720176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27947008 i_blocks 55305 00000020:00200000:2.0:1789760788.720180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.720191:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.720237:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27947008, count: 4096 00000080:00200000:2.0:1789760788.720245:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27947008, bytes: 4096 00000080:00200000:2.0:1789760788.720254:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.720262:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.720264:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27947008, 27951104) 0 0 00000080:00200000:2.0:1789760788.720270:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.720286:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6823, 6823] 00020000:00200000:2.0:1789760788.720295:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27947008 chunk: [27947008, 27951104) 27951104, 4096 00020000:00200000:2.0:1789760788.720299:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.720318:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6823, 6823] 00000008:00010000:2.0:1789760788.720335:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.720341:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.720353:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27947008, 27951104) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.720367:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27947008=0x1aa7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.720573:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.720576:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27951104 00000020:00010000:2.0:1789760788.720585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.720589:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.720593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.720597:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.720600:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.720608:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.720614:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.720630:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760788.720741:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 27947008<>27951104 00000100:00100000:3.0:1789760788.721238:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859456:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.764728:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974142 gen 3 00000008:00010000:3.0:1789760788.764794:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760788.764802:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.764815:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.764835:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760788.764866:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.764962:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6823, 6823] 00000080:00200000:2.0:1789760788.764986:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.764992:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.765007:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27951104 i_blocks 55305 00000020:00200000:2.0:1789760788.765011:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.765023:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.765157:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27951104, count: 4096 00000080:00200000:2.0:1789760788.765165:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27951104, bytes: 4096 00000080:00200000:2.0:1789760788.765176:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.765184:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.765187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27951104, 27955200) 0 0 00000080:00200000:2.0:1789760788.765191:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.765275:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6824, 6824] 00020000:00200000:2.0:1789760788.765284:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27951104 chunk: [27951104, 27955200) 27955200, 4096 00020000:00200000:2.0:1789760788.765289:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.765307:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6824, 6824] 00000008:00010000:2.0:1789760788.765325:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.765331:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.765344:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27951104, 27955200) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.765358:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27951104=0x1aa8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.765555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.765559:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27955200 00000020:00010000:2.0:1789760788.765567:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.765571:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.765575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.765578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.765581:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.765589:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.765596:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.765612:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760788.765709:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 27951104<>27955200 00000100:00100000:3.0:1789760788.765777:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859584:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760788.815922:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760788.815937:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.815948:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.815966:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760788.816004:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.816096:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6824, 6824] 00000080:00200000:2.0:1789760788.816111:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.816117:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.816132:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27955200 i_blocks 55305 00000020:00200000:2.0:1789760788.816136:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.816145:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.816181:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27955200, count: 4096 00000080:00200000:2.0:1789760788.816189:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27955200, bytes: 4096 00000080:00200000:2.0:1789760788.816208:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.816217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.816220:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27955200, 27959296) 0 0 00000080:00200000:2.0:1789760788.816225:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.816245:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6825, 6825] 00020000:00200000:2.0:1789760788.816254:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27955200 chunk: [27955200, 27959296) 27959296, 4096 00020000:00200000:2.0:1789760788.816259:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.816279:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6825, 6825] 00000008:00010000:2.0:1789760788.816298:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.816304:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.816317:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27955200, 27959296) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.816331:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27955200=0x1aa9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.816389:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.816565:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27959296 00000020:00010000:2.0:1789760788.816575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.816579:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.816582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.816586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.816588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.816596:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.816603:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.816619:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760788.816835:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 27955200<>27959296 00000100:00100000:3.0:1789760788.817007:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.871364:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974144 gen 3 00000008:00010000:3.0:1789760788.871472:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760788.871480:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.871491:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.871512:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760788.871540:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.871938:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6825, 6825] 00000080:00200000:2.0:1789760788.871954:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.871960:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.871985:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27959296 i_blocks 55305 00000020:00200000:2.0:1789760788.871990:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.872000:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.872034:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27959296, count: 4096 00000080:00200000:2.0:1789760788.872042:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27959296, bytes: 4096 00000080:00200000:2.0:1789760788.872052:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.872061:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.872063:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27959296, 27963392) 0 0 00000080:00200000:2.0:1789760788.872067:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.872083:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6826, 6826] 00020000:00200000:2.0:1789760788.872091:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27959296 chunk: [27959296, 27963392) 27963392, 4096 00020000:00200000:2.0:1789760788.872096:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.872115:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6826, 6826] 00000008:00010000:2.0:1789760788.872134:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.872141:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.872152:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27959296, 27963392) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.872166:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27959296=0x1aaa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.872221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.872224:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27963392 00000020:00010000:2.0:1789760788.872232:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.872235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.872239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.872242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.872245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.872252:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.872259:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.872273:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760788.872378:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 27959296<>27963392 00000100:00100000:3.0:1789760788.874275:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859840:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760788.910927:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974145 gen 3 00000008:00010000:3.0:1789760788.910988:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760788.910998:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.911013:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.911134:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760788.911164:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493859840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760788.911309:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6826, 6826] 00000080:00200000:2.0:1789760788.911326:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.911331:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.911347:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27963392 i_blocks 55305 00000020:00200000:2.0:1789760788.911350:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.911361:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.911525:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27963392, count: 4096 00000080:00200000:2.0:1789760788.911532:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27963392, bytes: 4096 00000080:00200000:2.0:1789760788.911544:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.911553:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.911556:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27963392, 27967488) 0 0 00000080:00200000:2.0:1789760788.911735:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.911753:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6827, 6827] 00020000:00200000:2.0:1789760788.911762:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27963392 chunk: [27963392, 27967488) 27967488, 4096 00020000:00200000:2.0:1789760788.911767:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.911787:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6827, 6827] 00000008:00010000:2.0:1789760788.911805:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.911811:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760788.911824:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27963392, 27967488) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.911838:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27963392=0x1aab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.911895:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.911898:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27967488 00000020:00010000:2.0:1789760788.911906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.911909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.911913:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.911916:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.911919:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.911927:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.911933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.911950:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760788.912163:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 27963392<>27967488 00000100:00100000:3.0:1789760788.912297:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859968:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.003248:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974146 gen 3 00000008:00010000:3.0:1789760789.003316:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760789.003325:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.003337:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.003361:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000080:00200000:2.0:1789760789.003636:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6827, 6827] 00000080:00200000:2.0:1789760789.003654:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.003660:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.003676:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27967488 i_blocks 55305 00000020:00200000:2.0:1789760789.003680:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.003691:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.003727:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27967488, count: 4096 00000080:00200000:2.0:1789760789.003736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27967488, bytes: 4096 00000080:00200000:2.0:1789760789.003749:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.003758:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.003760:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27967488, 27971584) 0 0 00000080:00200000:2.0:1789760789.003765:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.003783:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6828, 6828] 00020000:00200000:2.0:1789760789.003791:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27967488 chunk: [27967488, 27971584) 27971584, 4096 00020000:00200000:2.0:1789760789.003796:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.003818:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6828, 6828] 00000008:00010000:2.0:1789760789.003837:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.003844:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.003857:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27967488, 27971584) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.003873:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27967488=0x1aac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.003926:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.003929:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27971584 00000020:00010000:2.0:1789760789.003937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.003941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.003945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.003949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.003951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.003959:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.003966:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760789.004019:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493859968:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760789.004144:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:3.0:1789760789.004587:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 27967488<>27971584 00000100:00100000:3.0:1789760789.004625:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.004643:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.080489:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974147 gen 3 00000008:00010000:3.0:1789760789.080537:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760789.080547:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.080559:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.080575:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760789.080601:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860096:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.083184:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6828, 6828] 00000080:00200000:2.0:1789760789.083212:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.083219:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.083238:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27971584 i_blocks 55305 00000020:00200000:2.0:1789760789.083243:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.083261:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.083294:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27971584, count: 4096 00000080:00200000:2.0:1789760789.083303:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27971584, bytes: 4096 00000080:00200000:2.0:1789760789.083318:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.083327:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.083329:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27971584, 27975680) 0 0 00000080:00200000:2.0:1789760789.083334:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.083349:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6829, 6829] 00020000:00200000:2.0:1789760789.083361:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27971584 chunk: [27971584, 27975680) 27975680, 4096 00020000:00200000:2.0:1789760789.083366:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.083387:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6829, 6829] 00000008:00010000:2.0:1789760789.083715:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.083722:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.083734:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27971584, 27975680) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.083749:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27971584=0x1aad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.083811:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.083814:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27975680 00000020:00010000:2.0:1789760789.083941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.083945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.083948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.083952:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.083955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.083964:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.083971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.084000:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760789.085245:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 27971584<>27975680 00000100:00100000:3.0:1789760789.085340:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860224:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.148852:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974148 gen 3 00000008:00010000:3.0:1789760789.148889:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760789.148896:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.148906:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.148930:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760789.148956:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860224:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.149145:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6829, 6829] 00000080:00200000:2.0:1789760789.149157:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.149163:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.149174:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27975680 i_blocks 55305 00000020:00200000:2.0:1789760789.149178:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.149188:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.149216:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27975680, count: 4096 00000080:00200000:2.0:1789760789.149222:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27975680, bytes: 4096 00000080:00200000:2.0:1789760789.149232:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.149239:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.149241:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27975680, 27979776) 0 0 00000080:00200000:2.0:1789760789.149244:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.149258:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6830, 6830] 00020000:00200000:2.0:1789760789.149266:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27975680 chunk: [27975680, 27979776) 27979776, 4096 00020000:00200000:2.0:1789760789.149269:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.149284:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6830, 6830] 00000008:00010000:2.0:1789760789.149297:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.149302:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.149309:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27975680, 27979776) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.149321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27975680=0x1aae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.149362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.149365:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27979776 00000020:00010000:2.0:1789760789.149371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.149374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.149377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.149379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.149382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.149388:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.149440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.149464:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760789.149544:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27975680<>27979776 00000100:00100000:2.0:1789760789.149593:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860352:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.196497:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974149 gen 3 00000008:00010000:3.0:1789760789.196542:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760789.196548:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.196558:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.196583:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760789.196609:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860352:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760789.196814:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6830, 6830] 00000080:00200000:0.0:1789760789.196827:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.196833:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.196844:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27979776 i_blocks 55305 00000020:00200000:0.0:1789760789.196848:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.196858:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.196886:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27979776, count: 4096 00000080:00200000:0.0:1789760789.196892:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27979776, bytes: 4096 00000080:00200000:0.0:1789760789.196901:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.196908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.196910:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27979776, 27983872) 0 0 00000080:00200000:0.0:1789760789.196915:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.196929:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6831, 6831] 00020000:00200000:0.0:1789760789.196936:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27979776 chunk: [27979776, 27983872) 27983872, 4096 00020000:00200000:0.0:1789760789.196940:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.196955:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6831, 6831] 00000008:00010000:0.0:1789760789.196969:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.196984:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760789.197001:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27979776, 27983872) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.197012:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27979776=0x1aaf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.197162:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.197165:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27983872 00000020:00010000:0.0:1789760789.197172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.197176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.197179:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.197182:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.197185:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.197191:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.197197:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.197212:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760789.197294:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 27979776<>27983872 00000100:00100000:1.0:1789760789.197382:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493860480:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760789.247745:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760789.247757:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.247766:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.247844:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760789.247871:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493860480:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760789.247936:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6831, 6831] 00000080:00200000:3.0:1789760789.247949:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.247955:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.247967:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27983872 i_blocks 55305 00000020:00200000:3.0:1789760789.247970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.247991:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.248086:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27983872, count: 4096 00000080:00200000:3.0:1789760789.248093:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27983872, bytes: 4096 00000080:00200000:3.0:1789760789.248104:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.248112:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.248114:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27983872, 27987968) 0 0 00000080:00200000:3.0:1789760789.248119:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.248132:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6832, 6832] 00020000:00200000:3.0:1789760789.248140:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27983872 chunk: [27983872, 27987968) 27987968, 4096 00020000:00200000:3.0:1789760789.248144:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.248160:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6832, 6832] 00000008:00010000:3.0:1789760789.248173:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.248178:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760789.248187:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27983872, 27987968) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.248198:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27983872=0x1ab0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.248247:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.248250:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27987968 00000020:00010000:3.0:1789760789.248257:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.248261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.248264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.248267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.248269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.248276:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.248282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.248297:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760789.248377:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 27983872<>27987968 00000100:00100000:3.0:1789760789.248450:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760789.248463:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860608:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.318438:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974151 gen 3 00000008:00010000:3.0:1789760789.318484:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760789.318490:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.318498:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.318519:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760789.318543:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860608:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.318797:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6832, 6832] 00000080:00200000:2.0:1789760789.318810:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.318816:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.318828:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27987968 i_blocks 55305 00000020:00200000:2.0:1789760789.318832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.318840:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.318869:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27987968, count: 4096 00000080:00200000:2.0:1789760789.318874:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27987968, bytes: 4096 00000080:00200000:2.0:1789760789.318893:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.318900:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.318903:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27987968, 27992064) 0 0 00000080:00200000:2.0:1789760789.318907:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.318920:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6833, 6833] 00020000:00200000:2.0:1789760789.318927:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27987968 chunk: [27987968, 27992064) 27992064, 4096 00020000:00200000:2.0:1789760789.318932:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.318947:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6833, 6833] 00000008:00010000:2.0:1789760789.318966:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.318980:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.318990:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27987968, 27992064) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.319001:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27987968=0x1ab1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.319742:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.319748:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27992064 00000020:00010000:2.0:1789760789.319756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.319760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.319764:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.319767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.319769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.319777:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.319783:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.319802:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760789.319893:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27987968<>27992064 00000100:00100000:2.0:1789760789.319938:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760789.319956:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.369364:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974152 gen 3 00000008:00010000:3.0:1789760789.369435:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760789.369442:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.369451:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.369507:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760789.369532:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860736:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.369956:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6833, 6833] 00000080:00200000:2.0:1789760789.369970:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.370086:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.370099:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27992064 i_blocks 55305 00000020:00200000:2.0:1789760789.370103:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.370113:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.370140:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27992064, count: 4096 00000080:00200000:2.0:1789760789.370146:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27992064, bytes: 4096 00000080:00200000:2.0:1789760789.370155:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.370163:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.370165:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27992064, 27996160) 0 0 00000080:00200000:2.0:1789760789.370169:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.370182:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6834, 6834] 00020000:00200000:2.0:1789760789.370189:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27992064 chunk: [27992064, 27996160) 27996160, 4096 00020000:00200000:2.0:1789760789.370193:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.370336:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6834, 6834] 00000008:00010000:2.0:1789760789.370351:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.370356:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.370365:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27992064, 27996160) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.370376:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27992064=0x1ab2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.370453:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.370456:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27996160 00000020:00010000:2.0:1789760789.370463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.370466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.370469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.370472:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.370474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.370480:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.370486:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.370502:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760789.370579:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 27992064<>27996160 00000100:00100000:3.0:1789760789.371140:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860864:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.408618:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974153 gen 3 00000008:00010000:3.0:1789760789.408664:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760789.408670:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.408679:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.408693:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760789.408718:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493860864:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.408906:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6834, 6834] 00000080:00200000:2.0:1789760789.408919:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.408924:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.408935:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27996160 i_blocks 55305 00000020:00200000:2.0:1789760789.408938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.408947:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.408982:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27996160, count: 4096 00000080:00200000:2.0:1789760789.408989:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27996160, bytes: 4096 00000080:00200000:2.0:1789760789.408997:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.409004:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.409006:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27996160, 28000256) 0 0 00000080:00200000:2.0:1789760789.409010:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.409022:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6835, 6835] 00020000:00200000:2.0:1789760789.409029:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27996160 chunk: [27996160, 28000256) 28000256, 4096 00020000:00200000:2.0:1789760789.409033:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.409047:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6835, 6835] 00000008:00010000:2.0:1789760789.409060:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.409065:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.409073:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27996160, 28000256) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.409084:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=27996160=0x1ab3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.409127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.409130:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28000256 00000020:00010000:2.0:1789760789.409137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.409140:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.409143:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.409146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.409149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.409155:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.409161:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.409175:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760789.409249:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 27996160<>28000256 00000100:00100000:3.0:1789760789.409545:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860992:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.450752:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974154 gen 3 00000008:00010000:3.0:1789760789.450792:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760789.450800:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.450810:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.450873:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760789.450899:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493860992:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.451253:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6835, 6835] 00000080:00200000:2.0:1789760789.451267:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.451273:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.451284:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28000256 i_blocks 55305 00000020:00200000:2.0:1789760789.451287:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.451296:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.451385:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28000256, count: 4096 00000080:00200000:2.0:1789760789.451423:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28000256, bytes: 4096 00000080:00200000:2.0:1789760789.451433:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.451440:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.451442:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28000256, 28004352) 0 0 00000080:00200000:2.0:1789760789.451446:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.451526:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6836, 6836] 00020000:00200000:2.0:1789760789.451534:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28000256 chunk: [28000256, 28004352) 28004352, 4096 00020000:00200000:2.0:1789760789.451538:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.451710:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6836, 6836] 00000008:00010000:2.0:1789760789.451724:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.451728:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.451737:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28000256, 28004352) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.451747:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28000256=0x1ab4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.451794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.451797:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28004352 00000020:00010000:2.0:1789760789.451804:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.451807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.451809:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.451812:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.451814:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.451820:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.451825:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.451845:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760789.451927:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 28000256<>28004352 00000100:00100000:3.0:1789760789.452008:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493861120:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.497202:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974155 gen 3 00000008:00010000:3.0:1789760789.497245:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760789.497252:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.497263:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.497282:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760789.497307:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493861120:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.498246:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6836, 6836] 00000080:00200000:2.0:1789760789.498261:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.498266:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.498278:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28004352 i_blocks 55305 00000020:00200000:2.0:1789760789.498282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.498292:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.498331:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28004352, count: 4096 00000080:00200000:2.0:1789760789.498337:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28004352, bytes: 4096 00000080:00200000:2.0:1789760789.498347:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.498355:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.498357:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28004352, 28008448) 0 0 00000080:00200000:2.0:1789760789.498361:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.498373:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6837, 6837] 00020000:00200000:2.0:1789760789.498379:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28004352 chunk: [28004352, 28008448) 28008448, 4096 00020000:00200000:2.0:1789760789.498383:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.498438:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6837, 6837] 00000008:00010000:2.0:1789760789.498463:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.498467:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.498477:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28004352, 28008448) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.498488:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28004352=0x1ab5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.498535:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.498538:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28008448 00000020:00010000:2.0:1789760789.498546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.498549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.498552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.498554:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.498557:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.498563:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.498569:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.498741:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760789.498817:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 28004352<>28008448 00000100:00100000:3.0:1789760789.498886:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861248:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.536062:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974156 gen 3 00000008:00010000:3.0:1789760789.536113:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760789.536122:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.536135:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.536154:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760789.536185:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861248:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760789.536213:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6837, 6837] 00000080:00200000:2.0:1789760789.536233:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.536239:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.536254:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28008448 i_blocks 55305 00000020:00200000:2.0:1789760789.536260:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.536271:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.536299:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28008448, count: 4096 00000080:00200000:2.0:1789760789.536306:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28008448, bytes: 4096 00000080:00200000:2.0:1789760789.536314:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.536332:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.536335:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28008448, 28012544) 0 0 00000080:00200000:2.0:1789760789.536340:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.536353:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6838, 6838] 00020000:00200000:2.0:1789760789.536360:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28008448 chunk: [28008448, 28012544) 28012544, 4096 00020000:00200000:2.0:1789760789.536363:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.536379:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6838, 6838] 00000008:00010000:2.0:1789760789.536434:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.536439:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760789.536449:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28008448, 28012544) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.536459:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28008448=0x1ab6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.536510:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.536513:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28012544 00000020:00010000:2.0:1789760789.536520:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.536523:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.536526:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.536530:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.536533:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.536540:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.536546:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.536564:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760789.536777:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 28008448<>28012544 00000100:00100000:2.0:1789760789.536829:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493861376:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760789.566638:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760789.566649:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.566659:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.566686:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760789.566712:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493861376:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760789.566811:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6838, 6838] 00000080:00200000:0.0:1789760789.566823:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.566829:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.566840:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28012544 i_blocks 55305 00000020:00200000:0.0:1789760789.566844:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.566853:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.566879:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28012544, count: 4096 00000080:00200000:0.0:1789760789.566886:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28012544, bytes: 4096 00000080:00200000:0.0:1789760789.566894:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.566901:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.566903:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28012544, 28016640) 0 0 00000080:00200000:0.0:1789760789.566907:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.566920:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6839, 6839] 00020000:00200000:0.0:1789760789.566927:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28012544 chunk: [28012544, 28016640) 28016640, 4096 00020000:00200000:0.0:1789760789.566930:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.566946:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6839, 6839] 00000008:00010000:0.0:1789760789.566958:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.566963:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760789.566982:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28012544, 28016640) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.566993:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28012544=0x1ab7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.567035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.567038:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28016640 00000020:00010000:0.0:1789760789.567044:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.567048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.567050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.567053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.567056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.567062:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.567067:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.567082:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760789.567882:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 28012544<>28016640 00000100:00100000:0.0:1789760789.567940:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493861504:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760789.607313:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974158 gen 3 00000008:00010000:0.0:1789760789.607369:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760789.607375:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.607383:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.607431:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760789.607455:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493861504:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.607551:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6839, 6839] 00000080:00200000:1.0:1789760789.607562:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.607567:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.607578:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28016640 i_blocks 55305 00000020:00200000:1.0:1789760789.607581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.607588:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.607615:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28016640, count: 4096 00000080:00200000:1.0:1789760789.607621:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28016640, bytes: 4096 00000080:00200000:1.0:1789760789.607628:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.607636:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.607638:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28016640, 28020736) 0 0 00000080:00200000:1.0:1789760789.607641:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.607653:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6840, 6840] 00020000:00200000:1.0:1789760789.607659:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28016640 chunk: [28016640, 28020736) 28020736, 4096 00020000:00200000:1.0:1789760789.607664:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.607678:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6840, 6840] 00000008:00010000:1.0:1789760789.607690:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.607694:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.607703:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28016640, 28020736) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.607713:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28016640=0x1ab8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.607756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.607758:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28020736 00000020:00010000:1.0:1789760789.607765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.607768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.607771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.607774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.607776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.607783:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.607788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.607800:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760789.607875:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ca80 - object 0x240000400:5 offset 28016640<>28020736 00000100:00100000:1.0:1789760789.607913:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.607924:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493861632:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760789.640916:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:1.0:1789760789.640932:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.640943:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.641272:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:1.0:1789760789.641306:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493861632:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760789.641371:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6840, 6840] 00000080:00200000:3.0:1789760789.641384:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.641389:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.641428:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28020736 i_blocks 55305 00000020:00200000:3.0:1789760789.641431:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.641441:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.641473:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28020736, count: 4096 00000080:00200000:3.0:1789760789.641480:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28020736, bytes: 4096 00000080:00200000:3.0:1789760789.641491:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.641500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.641502:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28020736, 28024832) 0 0 00000080:00200000:3.0:1789760789.641507:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.641525:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6841, 6841] 00020000:00200000:3.0:1789760789.641533:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28020736 chunk: [28020736, 28024832) 28024832, 4096 00020000:00200000:3.0:1789760789.641537:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.641557:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6841, 6841] 00000008:00010000:3.0:1789760789.641574:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.641581:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760789.641593:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28020736, 28024832) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.641606:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28020736=0x1ab9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.641666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.641668:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28024832 00000020:00010000:3.0:1789760789.641676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.641680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.641683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.641687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.641690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.641698:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.641705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.641721:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760789.641834:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 28020736<>28024832 00000100:00100000:2.0:1789760789.641906:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861760:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760789.679586:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974160 gen 3 00000008:00010000:3.0:1789760789.680289:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760789.680298:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.680310:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.680481:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760789.680511:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861760:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.681184:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6841, 6841] 00000080:00200000:1.0:1789760789.681205:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.681214:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.681232:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28024832 i_blocks 55305 00000020:00200000:1.0:1789760789.681236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.681248:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.681283:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28024832, count: 4096 00000080:00200000:1.0:1789760789.681292:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28024832, bytes: 4096 00000080:00200000:1.0:1789760789.681307:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.681316:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.681318:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28024832, 28028928) 0 0 00000080:00200000:1.0:1789760789.681323:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.681340:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6842, 6842] 00020000:00200000:1.0:1789760789.681349:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28024832 chunk: [28024832, 28028928) 28028928, 4096 00020000:00200000:1.0:1789760789.681353:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.681374:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6842, 6842] 00000008:00010000:1.0:1789760789.681407:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.681414:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.681687:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28024832, 28028928) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.681703:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28024832=0x1aba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.681758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.681762:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28028928 00000020:00010000:1.0:1789760789.681771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.681774:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.681777:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.681781:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.681785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.681797:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.681805:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.681924:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:3.0:1789760789.682046:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 28024832<>28028928 00000100:00100000:2.0:1789760789.682134:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760789.682215:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861888:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760789.743272:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760789.751441:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:3.0:1789760789.751530:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.751544:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.751570:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:3.0:1789760789.751601:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493861888:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.753073:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6842, 6842] 00000080:00200000:1.0:1789760789.753097:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.753103:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.753120:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28028928 i_blocks 55305 00000020:00200000:1.0:1789760789.753126:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.753139:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.753175:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28028928, count: 4096 00000080:00200000:1.0:1789760789.753183:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28028928, bytes: 4096 00000080:00200000:1.0:1789760789.753196:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.753206:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.753208:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28028928, 28033024) 0 0 00000080:00200000:1.0:1789760789.753213:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.753230:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6843, 6843] 00020000:00200000:1.0:1789760789.753240:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28028928 chunk: [28028928, 28033024) 28033024, 4096 00020000:00200000:1.0:1789760789.753244:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.753263:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6843, 6843] 00000008:00010000:1.0:1789760789.753281:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.753287:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.753299:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28028928, 28033024) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.753314:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28028928=0x1abb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.753440:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.753444:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28033024 00000020:00010000:1.0:1789760789.753832:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.753839:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.753843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.753847:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.753850:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.753863:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.753871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.753895:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92300 00000008:00100000:1.0:1789760789.754036:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f100 - object 0x240000400:5 offset 28028928<>28033024 00000100:00100000:0.0:1789760789.754878:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862016:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760789.806476:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974162 gen 3 00000008:00010000:0.0:1789760789.806543:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92300 00000020:00200000:0.0:1789760789.806553:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.806563:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.806635:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92300 00000100:00100000:0.0:1789760789.806664:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862016:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.806787:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6843, 6843] 00000080:00200000:1.0:1789760789.806804:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.806811:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.806828:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28033024 i_blocks 55305 00000020:00200000:1.0:1789760789.806832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.806842:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.806875:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28033024, count: 4096 00000080:00200000:1.0:1789760789.806882:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28033024, bytes: 4096 00000080:00200000:1.0:1789760789.806892:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.806900:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.806902:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28033024, 28037120) 0 0 00000080:00200000:1.0:1789760789.806907:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.806923:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6844, 6844] 00020000:00200000:1.0:1789760789.806931:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28033024 chunk: [28033024, 28037120) 28037120, 4096 00020000:00200000:1.0:1789760789.806935:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.806954:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6844, 6844] 00000008:00010000:1.0:1789760789.807042:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.807048:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.807062:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28033024, 28037120) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.807076:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28033024=0x1abc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.807205:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.807209:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28037120 00000020:00010000:1.0:1789760789.807217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.807221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.807224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.807228:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.807231:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.807242:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.807249:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.807265:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760789.807363:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d180 - object 0x240000400:5 offset 28033024<>28037120 00000100:00100000:0.0:1789760789.807594:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.807613:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862144:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760789.862386:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974163 gen 3 00000008:00010000:0.0:1789760789.862537:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760789.862546:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.862558:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.862578:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760789.862608:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862144:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.862705:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6844, 6844] 00000080:00200000:1.0:1789760789.862722:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.862729:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.862745:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28037120 i_blocks 55305 00000020:00200000:1.0:1789760789.862750:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.862759:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.862792:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28037120, count: 4096 00000080:00200000:1.0:1789760789.862800:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28037120, bytes: 4096 00000080:00200000:1.0:1789760789.862810:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.862820:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.862822:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28037120, 28041216) 0 0 00000080:00200000:1.0:1789760789.862827:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.862843:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6845, 6845] 00020000:00200000:1.0:1789760789.862851:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28037120 chunk: [28037120, 28041216) 28041216, 4096 00020000:00200000:1.0:1789760789.862856:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.862874:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6845, 6845] 00000008:00010000:1.0:1789760789.862891:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.862897:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.862909:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28037120, 28041216) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.862922:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28037120=0x1abd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.862991:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.862994:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28041216 00000020:00010000:1.0:1789760789.863002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.863006:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.863009:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.863012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.863015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.863027:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.863033:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.863047:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760789.863221:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ed80 - object 0x240000400:5 offset 28037120<>28041216 00000100:00100000:0.0:1789760789.863328:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862272:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760789.922384:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974164 gen 3 00000008:00010000:0.0:1789760789.922467:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760789.922476:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.922488:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.922510:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760789.922539:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862272:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.922652:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6845, 6845] 00000080:00200000:1.0:1789760789.922669:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.922676:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.922701:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28041216 i_blocks 55305 00000020:00200000:1.0:1789760789.922705:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.922717:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.922750:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28041216, count: 4096 00000080:00200000:1.0:1789760789.922757:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28041216, bytes: 4096 00000080:00200000:1.0:1789760789.922767:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.922776:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.922779:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28041216, 28045312) 0 0 00000080:00200000:1.0:1789760789.922783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.922799:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6846, 6846] 00020000:00200000:1.0:1789760789.922809:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28041216 chunk: [28041216, 28045312) 28045312, 4096 00020000:00200000:1.0:1789760789.922813:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.922842:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6846, 6846] 00000008:00010000:1.0:1789760789.922860:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.922866:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.922877:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28041216, 28045312) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.922892:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28041216=0x1abe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.922952:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.922955:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28045312 00000020:00010000:1.0:1789760789.922964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.922976:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.922979:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.922983:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.922986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.922997:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.923005:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.923020:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760789.923123:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f480 - object 0x240000400:5 offset 28041216<>28045312 00000100:00100000:0.0:1789760789.923302:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862400:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760789.983662:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974165 gen 3 00000008:00010000:0.0:1789760789.983711:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760789.983719:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.983729:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.984313:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760789.984347:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862400:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760789.984557:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6846, 6846] 00000080:00200000:1.0:1789760789.984574:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.984580:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.984596:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28045312 i_blocks 55305 00000020:00200000:1.0:1789760789.984601:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.984612:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.984644:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28045312, count: 4096 00000080:00200000:1.0:1789760789.984651:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28045312, bytes: 4096 00000080:00200000:1.0:1789760789.984665:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.984674:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.984676:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28045312, 28049408) 0 0 00000080:00200000:1.0:1789760789.984680:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.984696:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6847, 6847] 00020000:00200000:1.0:1789760789.984704:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28045312 chunk: [28045312, 28049408) 28049408, 4096 00020000:00200000:1.0:1789760789.984709:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.984728:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6847, 6847] 00000008:00010000:1.0:1789760789.984807:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.984813:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760789.984825:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28045312, 28049408) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.984839:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28045312=0x1abf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.985605:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.985611:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28049408 00000020:00010000:1.0:1789760789.985623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.985627:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.985631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.985634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.985637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.985650:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.985657:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.985677:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760789.985915:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c000 - object 0x240000400:5 offset 28045312<>28049408 00000100:00100000:0.0:1789760789.986293:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862528:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760790.045647:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974166 gen 3 00000008:00010000:0.0:1789760790.045693:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760790.045700:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.045709:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.045726:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760790.045848:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862528:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760790.046283:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6847, 6847] 00000080:00200000:1.0:1789760790.046300:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.046307:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.046328:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28049408 i_blocks 55305 00000020:00200000:1.0:1789760790.046333:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.046344:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.046380:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28049408, count: 4096 00000080:00200000:1.0:1789760790.046388:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28049408, bytes: 4096 00000080:00200000:1.0:1789760790.046431:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.046440:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.046442:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28049408, 28053504) 0 0 00000080:00200000:1.0:1789760790.046559:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.046578:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6848, 6848] 00020000:00200000:1.0:1789760790.046587:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28049408 chunk: [28049408, 28053504) 28053504, 4096 00020000:00200000:1.0:1789760790.046591:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.046611:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6848, 6848] 00000008:00010000:1.0:1789760790.046629:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.046635:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760790.046647:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28049408, 28053504) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.046663:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28049408=0x1ac0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.046720:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.046723:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28053504 00000020:00010000:1.0:1789760790.046731:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.046734:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.046737:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.046860:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.046865:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.046877:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.046884:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.046902:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760790.047012:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016df80 - object 0x240000400:5 offset 28049408<>28053504 00000100:00100000:0.0:1789760790.047091:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862656:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760790.111262:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974167 gen 3 00000008:00010000:0.0:1789760790.111317:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760790.111326:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.111338:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.111368:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760790.111446:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862656:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760790.111731:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6848, 6848] 00000080:00200000:1.0:1789760790.111748:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.111754:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.111771:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28053504 i_blocks 55305 00000020:00200000:1.0:1789760790.111776:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.111787:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.111822:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28053504, count: 4096 00000080:00200000:1.0:1789760790.111830:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28053504, bytes: 4096 00000080:00200000:1.0:1789760790.111845:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.111855:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.111857:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28053504, 28057600) 0 0 00000080:00200000:1.0:1789760790.111862:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.111880:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6849, 6849] 00020000:00200000:1.0:1789760790.111889:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28053504 chunk: [28053504, 28057600) 28057600, 4096 00020000:00200000:1.0:1789760790.111893:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.111914:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6849, 6849] 00000008:00010000:1.0:1789760790.111934:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.111940:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760790.111951:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28053504, 28057600) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.111965:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28053504=0x1ac1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.112035:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.112038:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28057600 00000020:00010000:1.0:1789760790.112047:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.112050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.112053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.112057:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.112060:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.112071:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.112080:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.112096:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760790.112200:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c700 - object 0x240000400:5 offset 28053504<>28057600 00000100:00100000:0.0:1789760790.112267:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862784:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760790.157009:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974168 gen 3 00000008:00010000:0.0:1789760790.157058:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760790.157067:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.157078:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.157102:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000100:00100000:0.0:1789760790.157131:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493862784:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760790.157239:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6849, 6849] 00000080:00200000:1.0:1789760790.157255:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.157261:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.157277:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28057600 i_blocks 55305 00000020:00200000:1.0:1789760790.157281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.157291:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.157325:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28057600, count: 4096 00000080:00200000:1.0:1789760790.157333:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28057600, bytes: 4096 00000080:00200000:1.0:1789760790.157342:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.157350:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.157352:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28057600, 28061696) 0 0 00000080:00200000:1.0:1789760790.157357:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.157373:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6850, 6850] 00020000:00200000:1.0:1789760790.157381:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28057600 chunk: [28057600, 28061696) 28061696, 4096 00020000:00200000:1.0:1789760790.157385:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.157435:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6850, 6850] 00000008:00010000:1.0:1789760790.157453:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.157460:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760790.157472:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28057600, 28061696) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.157486:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28057600=0x1ac2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.157540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.157543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28061696 00000020:00010000:1.0:1789760790.157551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.157555:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.157559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.157562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.157564:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.157575:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.157582:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.157597:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760790.157695:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ce00 - object 0x240000400:5 offset 28057600<>28061696 00000100:00100000:0.0:1789760790.157761:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862912:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760790.200988:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760790.201004:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.201015:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.201035:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760790.201063:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493862912:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.201139:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6850, 6850] 00000080:00200000:2.0:1789760790.201156:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.201161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.201176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28061696 i_blocks 55305 00000020:00200000:2.0:1789760790.201180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.201189:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.201221:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28061696, count: 4096 00000080:00200000:2.0:1789760790.201227:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28061696, bytes: 4096 00000080:00200000:2.0:1789760790.201236:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.201244:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.201246:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28061696, 28065792) 0 0 00000080:00200000:2.0:1789760790.201250:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.201264:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6851, 6851] 00020000:00200000:2.0:1789760790.201273:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28061696 chunk: [28061696, 28065792) 28065792, 4096 00020000:00200000:2.0:1789760790.201278:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.201295:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6851, 6851] 00000008:00010000:2.0:1789760790.201312:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.201317:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.201328:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28061696, 28065792) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.201341:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28061696=0x1ac3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.201390:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.201429:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28065792 00000020:00010000:2.0:1789760790.201437:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.201440:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.201444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.201455:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.201458:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.201465:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.201471:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.201484:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760790.201588:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 28061696<>28065792 00000100:00100000:3.0:1789760790.201724:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.201742:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863040:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.244821:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974170 gen 3 00000008:00010000:3.0:1789760790.244890:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760790.244900:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.244911:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.244934:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760790.244964:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863040:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.245122:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6851, 6851] 00000080:00200000:2.0:1789760790.245140:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.245145:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.245160:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28065792 i_blocks 55305 00000020:00200000:2.0:1789760790.245163:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.245173:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.245207:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28065792, count: 4096 00000080:00200000:2.0:1789760790.245214:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28065792, bytes: 4096 00000080:00200000:2.0:1789760790.245223:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.245231:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.245233:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28065792, 28069888) 0 0 00000080:00200000:2.0:1789760790.245237:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.245253:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6852, 6852] 00020000:00200000:2.0:1789760790.245261:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28065792 chunk: [28065792, 28069888) 28069888, 4096 00020000:00200000:2.0:1789760790.245266:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.245286:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6852, 6852] 00000008:00010000:2.0:1789760790.245303:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.245309:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.245321:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28065792, 28069888) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.245335:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28065792=0x1ac4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.245435:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.245439:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28069888 00000020:00010000:2.0:1789760790.245457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.245461:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.245464:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.245467:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.245470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.245478:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.245484:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.245500:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760790.245600:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 28065792<>28069888 00000100:00100000:3.0:1789760790.245779:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.285684:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974171 gen 3 00000008:00010000:3.0:1789760790.285733:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760790.285741:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.285753:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.285772:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760790.285800:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863168:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.285906:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6852, 6852] 00000080:00200000:2.0:1789760790.285923:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.285928:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.285944:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28069888 i_blocks 55305 00000020:00200000:2.0:1789760790.285948:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.285958:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.285997:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28069888, count: 4096 00000080:00200000:2.0:1789760790.286006:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28069888, bytes: 4096 00000080:00200000:2.0:1789760790.286014:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.286028:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.286031:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28069888, 28073984) 0 0 00000080:00200000:2.0:1789760790.286035:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.286049:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6853, 6853] 00020000:00200000:2.0:1789760790.286057:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28069888 chunk: [28069888, 28073984) 28073984, 4096 00020000:00200000:2.0:1789760790.286062:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.286078:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6853, 6853] 00000008:00010000:2.0:1789760790.286093:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.286099:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.286109:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28069888, 28073984) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.286122:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28069888=0x1ac5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.286166:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.286168:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28073984 00000020:00010000:2.0:1789760790.286175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.286178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.286182:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.286185:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.286188:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.286194:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.286199:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.286311:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:3.0:1789760790.286424:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 28069888<>28073984 00000100:00100000:2.0:1789760790.286520:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760790.286764:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863296:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.327428:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974172 gen 3 00000008:00010000:3.0:1789760790.327485:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760790.327495:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.327507:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.327526:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760790.327558:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.327713:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6853, 6853] 00000080:00200000:2.0:1789760790.327730:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.327735:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.327752:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28073984 i_blocks 55305 00000020:00200000:2.0:1789760790.327756:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.327767:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.327799:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28073984, count: 4096 00000080:00200000:2.0:1789760790.327806:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28073984, bytes: 4096 00000080:00200000:2.0:1789760790.327815:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.327825:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.327827:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28073984, 28078080) 0 0 00000080:00200000:2.0:1789760790.327831:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.327848:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6854, 6854] 00020000:00200000:2.0:1789760790.327858:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28073984 chunk: [28073984, 28078080) 28078080, 4096 00020000:00200000:2.0:1789760790.327862:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.327882:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6854, 6854] 00000008:00010000:2.0:1789760790.327902:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.327908:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.327919:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28073984, 28078080) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.327932:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28073984=0x1ac6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.328000:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.328003:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28078080 00000020:00010000:2.0:1789760790.328012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.328015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.328027:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.328031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.328034:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.328043:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.328049:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.328064:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760790.328165:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 28073984<>28078080 00000100:00100000:2.0:1789760790.328210:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760790.328226:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863424:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.358265:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974173 gen 3 00000008:00010000:3.0:1789760790.358320:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760790.358330:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.358342:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.358364:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760790.358548:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863424:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.358690:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6854, 6854] 00000080:00200000:2.0:1789760790.358705:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.358711:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.358726:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28078080 i_blocks 55305 00000020:00200000:2.0:1789760790.358730:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.358740:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.358771:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28078080, count: 4096 00000080:00200000:2.0:1789760790.358777:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28078080, bytes: 4096 00000080:00200000:2.0:1789760790.358786:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.358795:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.358798:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28078080, 28082176) 0 0 00000080:00200000:2.0:1789760790.358802:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.358829:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6855, 6855] 00020000:00200000:2.0:1789760790.358837:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28078080 chunk: [28078080, 28082176) 28082176, 4096 00020000:00200000:2.0:1789760790.358841:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.358860:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6855, 6855] 00000008:00010000:2.0:1789760790.358878:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.358884:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.358897:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28078080, 28082176) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.358911:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28078080=0x1ac7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.358977:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.358980:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28082176 00000020:00010000:2.0:1789760790.358989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.358992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.358995:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.358998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.359001:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.359008:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.359120:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.359138:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760790.359253:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28078080<>28082176 00000100:00100000:3.0:1789760790.359322:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863552:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.398822:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974174 gen 3 00000008:00010000:3.0:1789760790.398879:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760790.398888:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.398900:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.398921:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000080:00200000:2.0:1789760790.399004:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6855, 6855] 00000080:00200000:2.0:1789760790.399022:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.399028:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.399044:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28082176 i_blocks 55305 00000020:00200000:2.0:1789760790.399048:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.399059:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.399093:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28082176, count: 4096 00000080:00200000:2.0:1789760790.399102:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28082176, bytes: 4096 00000080:00200000:2.0:1789760790.399110:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.399119:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.399123:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28082176, 28086272) 0 0 00000080:00200000:2.0:1789760790.399127:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.399143:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6856, 6856] 00000100:00100000:3.0:1789760790.399150:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863552:192.168.203.160@tcp:4:dd.0 00020000:00200000:2.0:1789760790.399152:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28082176 chunk: [28082176, 28086272) 28086272, 4096 00020000:00200000:2.0:1789760790.399156:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.399175:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6856, 6856] 00000008:00010000:2.0:1789760790.399192:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.399198:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.399209:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28082176, 28086272) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.399224:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28082176=0x1ac8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.399273:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.399275:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28086272 00000020:00010000:2.0:1789760790.399282:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.399285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.399287:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.399290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.399292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.399297:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.399303:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.399314:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760790.399441:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 28082176<>28086272 00000100:00100000:3.0:1789760790.399539:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.399555:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863680:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.427792:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974175 gen 3 00000008:00010000:3.0:1789760790.427845:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760790.427853:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.427864:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.427923:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760790.427953:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493863680:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760790.428193:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6856, 6856] 00000080:00200000:2.0:1789760790.428211:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.428217:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.428233:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28086272 i_blocks 55305 00000020:00200000:2.0:1789760790.428237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.428249:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.428284:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28086272, count: 4096 00000080:00200000:2.0:1789760790.428293:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28086272, bytes: 4096 00000080:00200000:2.0:1789760790.428303:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.428311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.428315:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28086272, 28090368) 0 0 00000080:00200000:2.0:1789760790.428320:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.428337:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6857, 6857] 00020000:00200000:2.0:1789760790.428346:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28086272 chunk: [28086272, 28090368) 28090368, 4096 00020000:00200000:2.0:1789760790.428351:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.428370:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6857, 6857] 00000008:00010000:2.0:1789760790.428388:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.428419:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760790.428431:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28086272, 28090368) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.428446:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28086272=0x1ac9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.428500:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.428503:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28090368 00000020:00010000:2.0:1789760790.428511:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.428514:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.428518:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.428521:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.428584:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.428593:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.428673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.428691:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760790.428787:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 28086272<>28090368 00000100:00100000:2.0:1789760790.428826:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760790.428844:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863808:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.488347:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974176 gen 3 00000008:00010000:3.0:1789760790.488412:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760790.488421:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.488434:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.488466:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760790.488497:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493863808:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.488697:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6857, 6857] 00000080:00200000:0.0:1789760790.488714:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.488721:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.488736:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28090368 i_blocks 55305 00000020:00200000:0.0:1789760790.488740:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.488751:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.488783:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28090368, count: 4096 00000080:00200000:0.0:1789760790.488790:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28090368, bytes: 4096 00000080:00200000:0.0:1789760790.488805:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.488814:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.488816:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28090368, 28094464) 0 0 00000080:00200000:0.0:1789760790.488820:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.488836:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6858, 6858] 00020000:00200000:0.0:1789760790.488844:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28090368 chunk: [28090368, 28094464) 28094464, 4096 00020000:00200000:0.0:1789760790.488850:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.488868:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6858, 6858] 00000008:00010000:0.0:1789760790.488887:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.488893:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.488904:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28090368, 28094464) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.488921:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28090368=0x1aca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.489240:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.489244:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28094464 00000020:00010000:0.0:1789760790.489252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.489256:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.489259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.489264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.489267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.489275:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.489282:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.489300:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760790.489440:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 28090368<>28094464 00000100:00100000:0.0:1789760790.489482:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.489502:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493863936:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760790.527988:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760790.528003:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.528015:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.528048:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760790.528079:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493863936:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760790.528293:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6858, 6858] 00000080:00200000:1.0:1789760790.528310:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.528317:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.528335:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28094464 i_blocks 55305 00000020:00200000:1.0:1789760790.528340:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.528351:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.528384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28094464, count: 4096 00000080:00200000:1.0:1789760790.528391:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28094464, bytes: 4096 00000080:00200000:1.0:1789760790.528428:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.528438:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.528441:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28094464, 28098560) 0 0 00000080:00200000:1.0:1789760790.528456:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.528475:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6859, 6859] 00020000:00200000:1.0:1789760790.528484:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28094464 chunk: [28094464, 28098560) 28098560, 4096 00020000:00200000:1.0:1789760790.528489:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.528510:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6859, 6859] 00000008:00010000:1.0:1789760790.528530:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.528536:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760790.528548:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28094464, 28098560) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.528563:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28094464=0x1acb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.528622:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.528625:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28098560 00000020:00010000:1.0:1789760790.528633:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.528637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.528640:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.528643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.528647:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.528658:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.528666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.528682:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760790.528788:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 28094464<>28098560 00000100:00100000:1.0:1789760790.528943:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.528961:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864064:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.581899:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974178 gen 3 00000008:00010000:1.0:1789760790.581978:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760790.581986:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.581996:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.582019:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760790.582047:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864064:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.582147:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6859, 6859] 00000080:00200000:0.0:1789760790.582164:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.582171:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.582187:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28098560 i_blocks 55305 00000020:00200000:0.0:1789760790.582191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.582201:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.582235:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28098560, count: 4096 00000080:00200000:0.0:1789760790.582242:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28098560, bytes: 4096 00000080:00200000:0.0:1789760790.582252:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.582261:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.582263:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28098560, 28102656) 0 0 00000080:00200000:0.0:1789760790.582268:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.582283:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6860, 6860] 00020000:00200000:0.0:1789760790.582291:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28098560 chunk: [28098560, 28102656) 28102656, 4096 00020000:00200000:0.0:1789760790.582296:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.582313:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6860, 6860] 00000008:00010000:0.0:1789760790.582330:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.582336:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.582347:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28098560, 28102656) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.582361:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28098560=0x1acc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.582462:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.582466:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28102656 00000020:00010000:0.0:1789760790.582474:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.582478:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.582481:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.582484:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.582488:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.582495:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.582503:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.582521:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760790.582630:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28098560<>28102656 00000100:00100000:0.0:1789760790.582676:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.582693:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864192:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760790.630977:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974179 gen 3 00000008:00010000:0.0:1789760790.631039:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760790.631048:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760790.631058:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760790.631081:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760790.631109:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864192:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760790.631277:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6860, 6860] 00000080:00200000:3.0:1789760790.631295:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.631302:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.631317:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28102656 i_blocks 55305 00000020:00200000:3.0:1789760790.631321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.631333:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.631366:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28102656, count: 4096 00000080:00200000:3.0:1789760790.631373:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28102656, bytes: 4096 00000080:00200000:3.0:1789760790.631384:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.631422:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760790.631424:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28102656, 28106752) 0 0 00000080:00200000:3.0:1789760790.631429:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.631518:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6861, 6861] 00020000:00200000:3.0:1789760790.631527:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28102656 chunk: [28102656, 28106752) 28106752, 4096 00020000:00200000:3.0:1789760790.631532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.631554:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6861, 6861] 00000008:00010000:3.0:1789760790.631573:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.631580:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760790.631592:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28102656, 28106752) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.631607:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28102656=0x1acd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.631666:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.631669:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28106752 00000020:00010000:3.0:1789760790.631677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.631681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.631684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.631688:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.631691:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.631700:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.631706:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.631723:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760790.631907:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 28102656<>28106752 00000100:00100000:2.0:1789760790.631990:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493864320:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760790.666540:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760790.666557:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.666567:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.666587:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760790.666614:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493864320:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.666750:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6861, 6861] 00000080:00200000:0.0:1789760790.666766:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.666772:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.666787:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28106752 i_blocks 55305 00000020:00200000:0.0:1789760790.666790:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.666799:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.666840:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28106752, count: 4096 00000080:00200000:0.0:1789760790.666848:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28106752, bytes: 4096 00000080:00200000:0.0:1789760790.666856:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.666864:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.666867:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28106752, 28110848) 0 0 00000080:00200000:0.0:1789760790.666872:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.666887:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6862, 6862] 00020000:00200000:0.0:1789760790.666894:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28106752 chunk: [28106752, 28110848) 28110848, 4096 00020000:00200000:0.0:1789760790.666898:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.666915:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6862, 6862] 00000008:00010000:0.0:1789760790.666931:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.666937:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.666948:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28106752, 28110848) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.666962:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28106752=0x1ace000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.667038:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.667041:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28110848 00000020:00010000:0.0:1789760790.667050:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.667053:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.667056:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.667059:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.667062:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.667069:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.667076:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.667092:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760790.667197:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28106752<>28110848 00000100:00100000:1.0:1789760790.667441:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864448:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.703594:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974180 gen 3 00000100:00100000:1.0:1789760790.703635:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304d180 x1876699493864448/t4294974181(4294974181) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760806 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760790.703687:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760790.703694:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.703704:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.703716:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760790.703738:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864448:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.703837:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6862, 6862] 00000080:00200000:0.0:1789760790.703853:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.703859:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.703874:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28110848 i_blocks 55305 00000020:00200000:0.0:1789760790.703878:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.703888:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.703924:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28110848, count: 4096 00000080:00200000:0.0:1789760790.703932:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28110848, bytes: 4096 00000080:00200000:0.0:1789760790.703942:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.703950:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.703952:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28110848, 28114944) 0 0 00000080:00200000:0.0:1789760790.703958:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.703989:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6863, 6863] 00020000:00200000:0.0:1789760790.703997:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28110848 chunk: [28110848, 28114944) 28114944, 4096 00020000:00200000:0.0:1789760790.704002:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.704030:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6863, 6863] 00000008:00010000:0.0:1789760790.704046:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.704052:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.704064:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28110848, 28114944) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.704077:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28110848=0x1acf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.704137:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.704140:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28114944 00000020:00010000:0.0:1789760790.704149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.704153:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.704156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.704159:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.704161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.704168:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.704174:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.704190:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760790.704288:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 28110848<>28114944 00000100:00100000:1.0:1789760790.704445:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.704463:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864576:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.751809:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974182 gen 3 00000008:00010000:1.0:1789760790.751878:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760790.751887:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.751896:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.751977:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760790.752007:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864576:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.753140:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6863, 6863] 00000080:00200000:0.0:1789760790.753162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.753169:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.753187:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28114944 i_blocks 55305 00000020:00200000:0.0:1789760790.753192:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.753210:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.753244:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28114944, count: 4096 00000080:00200000:0.0:1789760790.753252:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28114944, bytes: 4096 00000080:00200000:0.0:1789760790.753265:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.753275:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.753277:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28114944, 28119040) 0 0 00000080:00200000:0.0:1789760790.753283:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.753299:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6864, 6864] 00020000:00200000:0.0:1789760790.753308:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28114944 chunk: [28114944, 28119040) 28119040, 4096 00020000:00200000:0.0:1789760790.753312:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.753333:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6864, 6864] 00000008:00010000:0.0:1789760790.753351:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.753357:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.753370:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28114944, 28119040) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.753383:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28114944=0x1ad0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.753540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.753543:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28119040 00000020:00010000:0.0:1789760790.753553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.753556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.753559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.753563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.753566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.753575:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.753581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.753603:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760790.753714:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 28114944<>28119040 00000100:00100000:1.0:1789760790.753801:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864704:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760790.764946:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760790.787323:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760790.787340:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.787353:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.787378:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760790.787446:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493864704:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.787546:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6864, 6864] 00000080:00200000:0.0:1789760790.787563:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.787569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.787585:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28119040 i_blocks 55305 00000020:00200000:0.0:1789760790.787589:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.787599:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.787633:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28119040, count: 4096 00000080:00200000:0.0:1789760790.787641:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28119040, bytes: 4096 00000080:00200000:0.0:1789760790.787650:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.787659:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.787662:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28119040, 28123136) 0 0 00000080:00200000:0.0:1789760790.787667:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.787684:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6865, 6865] 00020000:00200000:0.0:1789760790.787693:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28119040 chunk: [28119040, 28123136) 28123136, 4096 00020000:00200000:0.0:1789760790.787697:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.787717:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6865, 6865] 00000008:00010000:0.0:1789760790.787734:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.787740:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.787752:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28119040, 28123136) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.787766:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28119040=0x1ad1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.788073:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.788076:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28123136 00000020:00010000:0.0:1789760790.788087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.788091:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.788094:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.788098:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.788101:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.788110:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.788117:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.788134:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760790.788246:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 28119040<>28123136 00000100:00100000:1.0:1789760790.788440:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864832:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.814992:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974184 gen 3 00000008:00010000:1.0:1789760790.815069:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760790.815077:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.815088:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.815114:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760790.815144:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864832:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.815361:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6865, 6865] 00000080:00200000:0.0:1789760790.815378:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.815384:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.815428:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28123136 i_blocks 55305 00000020:00200000:0.0:1789760790.815433:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.815453:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.815488:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28123136, count: 4096 00000080:00200000:0.0:1789760790.815495:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28123136, bytes: 4096 00000080:00200000:0.0:1789760790.815505:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.815514:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.815517:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28123136, 28127232) 0 0 00000080:00200000:0.0:1789760790.815522:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.815539:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6866, 6866] 00020000:00200000:0.0:1789760790.815548:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28123136 chunk: [28123136, 28127232) 28127232, 4096 00020000:00200000:0.0:1789760790.815553:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.815574:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6866, 6866] 00000008:00010000:0.0:1789760790.815590:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.815596:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.815608:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28123136, 28127232) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.815623:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28123136=0x1ad2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.815684:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.815687:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28127232 00000020:00010000:0.0:1789760790.815696:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.815699:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.815702:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.815706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.815708:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.815715:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.815723:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.815739:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760790.815980:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28123136<>28127232 00000100:00100000:1.0:1789760790.816148:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.816169:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864960:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.851345:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974185 gen 3 00000008:00010000:1.0:1789760790.851417:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760790.851426:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.851437:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.851505:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760790.851533:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493864960:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.852895:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6866, 6866] 00000080:00200000:0.0:1789760790.852916:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.852923:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.852940:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28127232 i_blocks 55305 00000020:00200000:0.0:1789760790.852946:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.852969:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.853006:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28127232, count: 4096 00000080:00200000:0.0:1789760790.853076:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28127232, bytes: 4096 00000080:00200000:0.0:1789760790.853091:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.853100:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.853102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28127232, 28131328) 0 0 00000080:00200000:0.0:1789760790.853107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.853124:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6867, 6867] 00020000:00200000:0.0:1789760790.853133:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28127232 chunk: [28127232, 28131328) 28131328, 4096 00020000:00200000:0.0:1789760790.853138:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.853159:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6867, 6867] 00000008:00010000:0.0:1789760790.853178:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.853185:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.853197:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28127232, 28131328) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.853211:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28127232=0x1ad3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.853273:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.853277:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28131328 00000020:00010000:0.0:1789760790.853285:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.853289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.853292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.853295:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.853298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.853306:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.853313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.853338:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760790.853756:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 28127232<>28131328 00000100:00100000:1.0:1789760790.854193:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493865088:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760790.888075:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760790.888090:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.888103:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.888126:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760790.888158:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493865088:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760790.888252:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6867, 6867] 00000080:00200000:3.0:1789760790.888268:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760790.888274:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760790.888290:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28131328 i_blocks 55305 00000020:00200000:3.0:1789760790.888293:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760790.888304:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760790.888337:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28131328, count: 4096 00000080:00200000:3.0:1789760790.888345:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28131328, bytes: 4096 00000080:00200000:3.0:1789760790.888355:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760790.888364:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760790.888367:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28131328, 28135424) 0 0 00000080:00200000:3.0:1789760790.888371:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760790.888387:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6868, 6868] 00020000:00200000:3.0:1789760790.888429:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28131328 chunk: [28131328, 28135424) 28135424, 4096 00020000:00200000:3.0:1789760790.888435:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760790.888466:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6868, 6868] 00000008:00010000:3.0:1789760790.888486:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760790.888492:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760790.888504:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28131328, 28135424) append 0 append lockpos 0 00000080:00200000:3.0:1789760790.888518:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28131328=0x1ad4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760790.888572:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760790.888574:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28135424 00000020:00010000:3.0:1789760790.888582:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760790.888585:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.888588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760790.888591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760790.888594:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760790.888601:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760790.888608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760790.888624:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760790.888723:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 28131328<>28135424 00000100:00100000:3.0:1789760790.888767:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.888783:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493865216:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760790.913485:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974186 gen 3 00000100:00100000:3.0:1789760790.913531:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514aa00 x1876699493865216/t4294974187(4294974187) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760806 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760790.913581:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760790.913589:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.913598:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.913622:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760790.913647:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493865216:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760790.913788:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6868, 6868] 00000080:00200000:1.0:1789760790.913803:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760790.913808:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760790.913823:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28135424 i_blocks 55305 00000020:00200000:1.0:1789760790.913827:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760790.913837:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760790.913868:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28135424, count: 4096 00000080:00200000:1.0:1789760790.913875:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28135424, bytes: 4096 00000080:00200000:1.0:1789760790.913885:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760790.913893:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760790.913895:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28135424, 28139520) 0 0 00000080:00200000:1.0:1789760790.913899:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760790.913916:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6869, 6869] 00020000:00200000:1.0:1789760790.913924:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28135424 chunk: [28135424, 28139520) 28139520, 4096 00020000:00200000:1.0:1789760790.913928:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760790.913947:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6869, 6869] 00000008:00010000:1.0:1789760790.913976:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760790.913982:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760790.913993:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28135424, 28139520) append 0 append lockpos 0 00000080:00200000:1.0:1789760790.914008:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28135424=0x1ad5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760790.914066:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760790.914069:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28139520 00000020:00010000:1.0:1789760790.914077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760790.914081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.914084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760790.914087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760790.914090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760790.914101:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760790.914108:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760790.914124:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760790.914226:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 28135424<>28139520 00000100:00100000:1.0:1789760790.914273:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.914290:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493865344:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760790.953305:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974188 gen 3 00000008:00010000:1.0:1789760790.953356:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760790.953363:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.953371:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.953431:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760790.953563:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493865344:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760790.953792:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6869, 6869] 00000080:00200000:0.0:1789760790.953804:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.953809:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.953821:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28139520 i_blocks 55305 00000020:00200000:0.0:1789760790.953825:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.953833:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.953859:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28139520, count: 4096 00000080:00200000:0.0:1789760790.953864:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28139520, bytes: 4096 00000080:00200000:0.0:1789760790.954094:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.954102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.954105:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28139520, 28143616) 0 0 00000080:00200000:0.0:1789760790.954109:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.954122:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6870, 6870] 00020000:00200000:0.0:1789760790.954129:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28139520 chunk: [28139520, 28143616) 28143616, 4096 00020000:00200000:0.0:1789760790.954132:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.954147:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6870, 6870] 00000008:00010000:0.0:1789760790.954158:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.954164:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760790.954172:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28139520, 28143616) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.954183:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28139520=0x1ad6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.954229:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.954231:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28143616 00000020:00010000:0.0:1789760790.954238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.954241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.954244:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.954247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.954250:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.954256:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.954262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.954277:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760790.954353:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 28139520<>28143616 00000100:00100000:0.0:1789760790.954591:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760790.954606:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865472:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760791.020102:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974189 gen 3 00000008:00010000:0.0:1789760791.020138:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760791.020145:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.020154:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.020182:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760791.020210:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865472:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760791.020345:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6870, 6870] 00000080:00200000:3.0:1789760791.020358:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.020364:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.020376:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28143616 i_blocks 55305 00000020:00200000:3.0:1789760791.020379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.020389:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.020594:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28143616, count: 4096 00000080:00200000:3.0:1789760791.020604:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28143616, bytes: 4096 00000080:00200000:3.0:1789760791.020618:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.020627:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760791.020630:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28143616, 28147712) 0 0 00000080:00200000:3.0:1789760791.020635:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.020653:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6871, 6871] 00020000:00200000:3.0:1789760791.020662:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28143616 chunk: [28143616, 28147712) 28147712, 4096 00020000:00200000:3.0:1789760791.020667:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.020686:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6871, 6871] 00000008:00010000:3.0:1789760791.020702:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.020708:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760791.020721:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28143616, 28147712) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.020738:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28143616=0x1ad7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.020935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.020938:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28147712 00000020:00010000:3.0:1789760791.020945:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.020948:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.020951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.020954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.020957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.020973:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.020979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.020995:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760791.021070:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 28143616<>28147712 00000100:00100000:3.0:1789760791.021115:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760791.021130:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493865600:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.086698:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974190 gen 3 00000008:00010000:3.0:1789760791.086753:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760791.086762:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.086772:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.086792:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760791.086832:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493865600:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.089889:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6871, 6871] 00000080:00200000:0.0:1789760791.089912:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.089918:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.089931:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28147712 i_blocks 55305 00000020:00200000:0.0:1789760791.089935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.089947:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.089985:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28147712, count: 4096 00000080:00200000:0.0:1789760791.089991:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28147712, bytes: 4096 00000080:00200000:0.0:1789760791.090002:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.090009:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.090011:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28147712, 28151808) 0 0 00000080:00200000:0.0:1789760791.090014:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.090028:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6872, 6872] 00020000:00200000:0.0:1789760791.090036:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28147712 chunk: [28147712, 28151808) 28151808, 4096 00020000:00200000:0.0:1789760791.090039:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.090067:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6872, 6872] 00000008:00010000:0.0:1789760791.090082:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.090087:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760791.090097:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28147712, 28151808) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.090107:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28147712=0x1ad8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.090156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.090159:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28151808 00000020:00010000:0.0:1789760791.090165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.090168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.090171:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97758/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.090174:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.090176:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.090182:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.090187:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.090203:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760791.090287:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 28147712<>28151808 00000100:00100000:1.0:1789760791.090436:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865728:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760791.136723:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974191 gen 3 00000008:00010000:1.0:1789760791.136762:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760791.136768:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.136777:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.136798:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760791.136833:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865728:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.137039:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6872, 6872] 00000080:00200000:0.0:1789760791.137776:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.137784:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.139812:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28151808 i_blocks 55305 00000020:00200000:1.0:1789760791.139823:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.139837:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.139867:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28151808, count: 4096 00000080:00200000:1.0:1789760791.139874:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28151808, bytes: 4096 00000080:00200000:1.0:1789760791.139886:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.139894:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.139896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28151808, 28155904) 0 0 00000080:00200000:1.0:1789760791.139900:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.139913:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6873, 6873] 00020000:00200000:1.0:1789760791.139922:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28151808 chunk: [28151808, 28155904) 28155904, 4096 00020000:00200000:1.0:1789760791.139925:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.139941:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6873, 6873] 00000008:00010000:1.0:1789760791.139955:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.140886:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.140905:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28151808, 28155904) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.140917:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28151808=0x1ad9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.140984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.140987:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28155904 00000020:00010000:1.0:1789760791.140994:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.140997:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.141000:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.141003:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.141005:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.141015:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.141020:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.141086:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760791.141172:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 28151808<>28155904 00000100:00100000:1.0:1789760791.141226:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760791.141245:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865856:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760791.189580:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974192 gen 3 00000008:00010000:1.0:1789760791.189617:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:1.0:1789760791.189623:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.189631:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.189706:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:1.0:1789760791.189729:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493865856:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760791.191942:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6873, 6873] 00000080:00200000:3.0:1789760791.191972:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.191978:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.191991:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28155904 i_blocks 55305 00000020:00200000:3.0:1789760791.191995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.192006:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.192033:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28155904, count: 4096 00000080:00200000:3.0:1789760791.192040:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28155904, bytes: 4096 00000080:00200000:3.0:1789760791.192110:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.192119:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760791.192121:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28155904, 28160000) 0 0 00000080:00200000:3.0:1789760791.192125:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.192138:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6874, 6874] 00020000:00200000:3.0:1789760791.192145:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28155904 chunk: [28155904, 28160000) 28160000, 4096 00020000:00200000:3.0:1789760791.192148:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.192165:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6874, 6874] 00000008:00010000:3.0:1789760791.192178:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.192184:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760791.192193:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28155904, 28160000) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.192205:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28155904=0x1ada000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.192261:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.192264:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28160000 00000020:00010000:3.0:1789760791.192271:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.192274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.192277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.192281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.192283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.192290:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.192295:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.192322:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760791.192427:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 28155904<>28160000 00000100:00100000:2.0:1789760791.192644:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760791.192664:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493865984:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.237349:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974193 gen 3 00000008:00010000:3.0:1789760791.237385:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760791.237392:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.237617:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.237697:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760791.237723:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493865984:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760791.238083:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6874, 6874] 00000080:00200000:1.0:1789760791.238094:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.238100:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.238111:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28160000 i_blocks 55305 00000020:00200000:1.0:1789760791.238114:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.238123:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.238150:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28160000, count: 4096 00000080:00200000:1.0:1789760791.238157:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28160000, bytes: 4096 00000080:00200000:1.0:1789760791.238167:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.238174:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.238176:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28160000, 28164096) 0 0 00000080:00200000:1.0:1789760791.238180:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.238194:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6875, 6875] 00020000:00200000:1.0:1789760791.238201:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28160000 chunk: [28160000, 28164096) 28164096, 4096 00020000:00200000:1.0:1789760791.238205:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.238221:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6875, 6875] 00000008:00010000:1.0:1789760791.238699:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.238705:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.238716:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28160000, 28164096) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.238727:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28160000=0x1adb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.238776:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.238779:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28164096 00000020:00010000:1.0:1789760791.238785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.238789:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.238792:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.238795:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.238798:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.238807:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.238813:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.238833:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760791.238994:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016dc00 - object 0x240000400:5 offset 28160000<>28164096 00000100:00100000:0.0:1789760791.239070:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493866112:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760791.312496:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974194 gen 3 00000008:00010000:0.0:1789760791.312549:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760791.312556:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.312564:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.312588:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760791.312614:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493866112:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760791.312801:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6875, 6875] 00000080:00200000:1.0:1789760791.312816:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.312821:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.312834:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28164096 i_blocks 55305 00000020:00200000:1.0:1789760791.312837:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.312847:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.312876:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28164096, count: 4096 00000080:00200000:1.0:1789760791.312884:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28164096, bytes: 4096 00000080:00200000:1.0:1789760791.312896:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.312905:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.312907:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28164096, 28168192) 0 0 00000080:00200000:1.0:1789760791.312912:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.312927:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6876, 6876] 00020000:00200000:1.0:1789760791.312934:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28164096 chunk: [28164096, 28168192) 28168192, 4096 00020000:00200000:1.0:1789760791.312938:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.312953:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6876, 6876] 00000008:00010000:1.0:1789760791.312980:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.312985:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.312995:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28164096, 28168192) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.313009:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28164096=0x1adc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.313061:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.313064:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28168192 00000020:00010000:1.0:1789760791.313071:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.313073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.313076:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.313079:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.313081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.313090:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.313096:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.313110:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760791.313203:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 28164096<>28168192 00000100:00100000:0.0:1789760791.314107:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493866240:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760791.373107:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760791.373118:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.373127:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.373140:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760791.373168:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493866240:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760791.374262:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6876, 6876] 00000080:00200000:1.0:1789760791.374277:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.374283:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.374296:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28168192 i_blocks 55305 00000020:00200000:1.0:1789760791.374301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.374311:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.374340:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28168192, count: 4096 00000080:00200000:1.0:1789760791.374347:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28168192, bytes: 4096 00000080:00200000:1.0:1789760791.374358:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.374365:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.374367:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28168192, 28172288) 0 0 00000080:00200000:1.0:1789760791.374372:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.374385:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6877, 6877] 00020000:00200000:1.0:1789760791.374416:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28168192 chunk: [28168192, 28172288) 28172288, 4096 00020000:00200000:1.0:1789760791.374420:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.374436:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6877, 6877] 00000008:00010000:1.0:1789760791.374451:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.374455:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.374464:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28168192, 28172288) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.374475:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28168192=0x1add000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.374525:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.374528:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28172288 00000020:00010000:1.0:1789760791.374535:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.374538:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.374541:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.374545:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.374548:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.374557:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.374562:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.374578:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760791.374662:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 28168192<>28172288 00000100:00100000:1.0:1789760791.374710:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.374732:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493866368:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760791.405851:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974196 gen 3 00000008:00010000:1.0:1789760791.405897:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760791.405904:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.405912:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.405936:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760791.405969:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493866368:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760791.406956:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6877, 6877] 00000080:00200000:3.0:1789760791.406978:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760791.406984:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760791.406996:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28172288 i_blocks 55305 00000020:00200000:3.0:1789760791.406999:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760791.407008:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760791.407034:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28172288, count: 4096 00000080:00200000:3.0:1789760791.407040:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28172288, bytes: 4096 00000080:00200000:3.0:1789760791.407050:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760791.407057:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760791.407059:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28172288, 28176384) 0 0 00000080:00200000:3.0:1789760791.407063:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760791.407077:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6878, 6878] 00020000:00200000:3.0:1789760791.407084:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28172288 chunk: [28172288, 28176384) 28176384, 4096 00020000:00200000:3.0:1789760791.407088:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760791.407103:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6878, 6878] 00000008:00010000:3.0:1789760791.407117:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760791.407121:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760791.407130:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28172288, 28176384) append 0 append lockpos 0 00000080:00200000:3.0:1789760791.407140:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28172288=0x1ade000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760791.407186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760791.407189:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28176384 00000020:00010000:3.0:1789760791.407195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760791.407199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.407201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760791.407205:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760791.407207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760791.407212:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760791.407218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760791.407232:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760791.408862:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 28172288<>28176384 00000100:00100000:2.0:1789760791.408925:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760791.410851:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493866496:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760791.453691:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760791.453703:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.453712:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.453734:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760791.453761:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493866496:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.454169:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6878, 6878] 00000080:00200000:2.0:1789760791.454181:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.454186:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.454196:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28176384 i_blocks 55305 00000020:00200000:2.0:1789760791.454200:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.454208:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.454233:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28176384, count: 4096 00000080:00200000:2.0:1789760791.454239:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28176384, bytes: 4096 00000080:00200000:2.0:1789760791.454248:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.454255:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.454257:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28176384, 28180480) 0 0 00000080:00200000:2.0:1789760791.454261:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.454273:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6879, 6879] 00020000:00200000:2.0:1789760791.454279:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28176384 chunk: [28176384, 28180480) 28180480, 4096 00020000:00200000:2.0:1789760791.454283:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.454298:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6879, 6879] 00000008:00010000:2.0:1789760791.454310:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.454315:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.454322:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28176384, 28180480) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.454333:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28176384=0x1adf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.454376:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.454379:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28180480 00000020:00010000:2.0:1789760791.454385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.454388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.454391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.454414:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.454417:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.454423:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.454428:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.454503:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760791.454574:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 28176384<>28180480 00000100:00100000:3.0:1789760791.454809:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.454824:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866624:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.500223:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974198 gen 3 00000008:00010000:3.0:1789760791.500280:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760791.500287:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.500298:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.500391:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760791.500434:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866624:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.503350:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6879, 6879] 00000080:00200000:2.0:1789760791.503374:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.503380:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.503412:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28180480 i_blocks 55305 00000020:00200000:2.0:1789760791.503418:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.503435:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.503465:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28180480, count: 4096 00000080:00200000:2.0:1789760791.503475:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28180480, bytes: 4096 00000080:00200000:2.0:1789760791.503489:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.503498:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.503500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28180480, 28184576) 0 0 00000080:00200000:2.0:1789760791.503504:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.503519:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6880, 6880] 00020000:00200000:2.0:1789760791.503528:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28180480 chunk: [28180480, 28184576) 28184576, 4096 00020000:00200000:2.0:1789760791.503532:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.503550:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6880, 6880] 00000008:00010000:2.0:1789760791.503565:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.503570:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.503578:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28180480, 28184576) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.503589:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28180480=0x1ae0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.503648:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.503651:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28184576 00000020:00010000:2.0:1789760791.503657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.503660:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.503663:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.503666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.503669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.503676:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.503681:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.503709:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760791.503790:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 28180480<>28184576 00000100:00100000:3.0:1789760791.504047:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866752:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760791.546635:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760791.546652:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.546666:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.546685:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760791.546715:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866752:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.546953:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6880, 6880] 00000080:00200000:2.0:1789760791.546981:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.546987:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.547002:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28184576 i_blocks 55305 00000020:00200000:2.0:1789760791.547006:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.547082:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.547120:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28184576, count: 4096 00000080:00200000:2.0:1789760791.547128:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28184576, bytes: 4096 00000080:00200000:2.0:1789760791.547139:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.547148:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.547150:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28184576, 28188672) 0 0 00000080:00200000:2.0:1789760791.547155:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.547171:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6881, 6881] 00020000:00200000:2.0:1789760791.547180:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28184576 chunk: [28184576, 28188672) 28188672, 4096 00020000:00200000:2.0:1789760791.547185:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.547204:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6881, 6881] 00000008:00010000:2.0:1789760791.547223:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.547228:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.547240:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28184576, 28188672) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.547255:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28184576=0x1ae1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.547314:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.547318:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28188672 00000020:00010000:2.0:1789760791.547327:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.547331:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.547334:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.547338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.547341:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.547349:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.547356:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.547373:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760791.547665:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 28184576<>28188672 00000100:00100000:3.0:1789760791.547767:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.547789:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866880:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.590710:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974200 gen 3 00000008:00010000:3.0:1789760791.590774:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760791.590783:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.590794:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.590810:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760791.590850:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493866880:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.591254:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6881, 6881] 00000080:00200000:2.0:1789760791.591276:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.591282:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.591298:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28188672 i_blocks 55305 00000020:00200000:2.0:1789760791.591303:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.591314:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.591347:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28188672, count: 4096 00000080:00200000:2.0:1789760791.591355:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28188672, bytes: 4096 00000080:00200000:2.0:1789760791.591365:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.591374:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.591377:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28188672, 28192768) 0 0 00000080:00200000:2.0:1789760791.591381:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.591419:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6882, 6882] 00020000:00200000:2.0:1789760791.591429:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28188672 chunk: [28188672, 28192768) 28192768, 4096 00020000:00200000:2.0:1789760791.591432:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.591452:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6882, 6882] 00000008:00010000:2.0:1789760791.591470:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.591476:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.591488:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28188672, 28192768) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.591501:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28188672=0x1ae2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.591794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.591797:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28192768 00000020:00010000:2.0:1789760791.591807:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.591870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.591874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.591878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.591881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.591890:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.591898:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.591917:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760791.592092:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 28188672<>28192768 00000100:00100000:3.0:1789760791.592172:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493867008:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760791.632304:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760791.632322:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.632336:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.632359:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760791.632390:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493867008:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.632533:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6882, 6882] 00000080:00200000:0.0:1789760791.632550:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.632556:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.632571:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28192768 i_blocks 55305 00000020:00200000:0.0:1789760791.632575:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.632586:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.632621:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28192768, count: 4096 00000080:00200000:0.0:1789760791.632629:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28192768, bytes: 4096 00000080:00200000:0.0:1789760791.632638:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.632648:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.632650:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28192768, 28196864) 0 0 00000080:00200000:0.0:1789760791.632654:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.632670:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6883, 6883] 00020000:00200000:0.0:1789760791.632679:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28192768 chunk: [28192768, 28196864) 28196864, 4096 00020000:00200000:0.0:1789760791.632684:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.632703:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6883, 6883] 00000008:00010000:0.0:1789760791.632720:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.632726:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760791.632738:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28192768, 28196864) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.632753:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28192768=0x1ae3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.632813:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.632828:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28196864 00000020:00010000:0.0:1789760791.632837:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.632840:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.632843:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.632846:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.632849:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.632856:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.632862:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.632879:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760791.632997:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 28192768<>28196864 00000100:00100000:0.0:1789760791.633152:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.633169:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867136:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760791.674709:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974202 gen 3 00000008:00010000:0.0:1789760791.674774:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760791.674783:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.674794:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.674828:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760791.674857:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867136:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760791.674967:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6883, 6883] 00000080:00200000:1.0:1789760791.674983:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.674989:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.675005:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28196864 i_blocks 55305 00000020:00200000:1.0:1789760791.675009:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.675028:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.675063:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28196864, count: 4096 00000080:00200000:1.0:1789760791.675070:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28196864, bytes: 4096 00000080:00200000:1.0:1789760791.675079:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.675088:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.675091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28196864, 28200960) 0 0 00000080:00200000:1.0:1789760791.675095:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.675114:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6884, 6884] 00020000:00200000:1.0:1789760791.675123:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28196864 chunk: [28196864, 28200960) 28200960, 4096 00020000:00200000:1.0:1789760791.675128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.675147:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6884, 6884] 00000008:00010000:1.0:1789760791.675165:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.675171:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.675183:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28196864, 28200960) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.675197:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28196864=0x1ae4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.675258:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.675261:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28200960 00000020:00010000:1.0:1789760791.675270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.675273:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.675277:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.675280:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.675284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.675294:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.675301:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.675317:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760791.675461:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 28196864<>28200960 00000100:00100000:1.0:1789760791.675492:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.675508:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867264:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760791.731023:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760791.731040:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.731054:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.731087:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760791.731126:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867264:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.731562:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6884, 6884] 00000080:00200000:0.0:1789760791.731580:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.731587:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.731604:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28200960 i_blocks 55305 00000020:00200000:0.0:1789760791.731608:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.731619:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.731655:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28200960, count: 4096 00000080:00200000:0.0:1789760791.731663:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28200960, bytes: 4096 00000080:00200000:0.0:1789760791.731674:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.731683:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.731685:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28200960, 28205056) 0 0 00000080:00200000:0.0:1789760791.731690:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.731706:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6885, 6885] 00020000:00200000:0.0:1789760791.731714:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28200960 chunk: [28200960, 28205056) 28205056, 4096 00020000:00200000:0.0:1789760791.731718:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.731739:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6885, 6885] 00000008:00010000:0.0:1789760791.731758:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.731765:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760791.731776:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28200960, 28205056) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.731791:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28200960=0x1ae5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.731864:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.731867:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28205056 00000020:00010000:0.0:1789760791.731876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.731879:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.731882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.731886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.731890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.731897:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.731904:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.731921:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760791.732113:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 28200960<>28205056 00000100:00100000:1.0:1789760791.732187:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867392:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760791.769474:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974204 gen 3 00000008:00010000:1.0:1789760791.769543:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760791.769552:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.769562:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.769585:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760791.769615:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867392:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.769772:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6885, 6885] 00000080:00200000:0.0:1789760791.769791:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.769796:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.769814:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28205056 i_blocks 55305 00000020:00200000:0.0:1789760791.769828:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.769839:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.769872:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28205056, count: 4096 00000080:00200000:0.0:1789760791.769880:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28205056, bytes: 4096 00000080:00200000:0.0:1789760791.769890:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.769900:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.769902:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28205056, 28209152) 0 0 00000080:00200000:0.0:1789760791.769907:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.769923:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6886, 6886] 00020000:00200000:0.0:1789760791.769932:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28205056 chunk: [28205056, 28209152) 28209152, 4096 00020000:00200000:0.0:1789760791.769937:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.769967:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6886, 6886] 00000008:00010000:0.0:1789760791.769984:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.769991:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760791.770003:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28205056, 28209152) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.770027:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28205056=0x1ae6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.770081:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.770084:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28209152 00000020:00010000:0.0:1789760791.770110:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.770113:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.770116:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.770119:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.770122:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.770129:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.770136:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.770150:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760791.770246:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 28205056<>28209152 00000100:00100000:1.0:1789760791.770808:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867520:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760791.788648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760791.808380:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974205 gen 3 00000008:00010000:1.0:1789760791.808472:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760791.808481:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.808493:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.808515:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760791.808543:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867520:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760791.808942:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6886, 6886] 00000080:00200000:0.0:1789760791.808970:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.808976:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.808993:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28209152 i_blocks 55305 00000020:00200000:0.0:1789760791.808997:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.809008:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.809051:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28209152, count: 4096 00000080:00200000:0.0:1789760791.809059:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28209152, bytes: 4096 00000080:00200000:0.0:1789760791.809069:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.809078:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.809080:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28209152, 28213248) 0 0 00000080:00200000:0.0:1789760791.809085:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.809102:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6887, 6887] 00020000:00200000:0.0:1789760791.809110:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28209152 chunk: [28209152, 28213248) 28213248, 4096 00020000:00200000:0.0:1789760791.809115:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.809135:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6887, 6887] 00000008:00010000:0.0:1789760791.809153:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.809159:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760791.809171:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28209152, 28213248) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.809185:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28209152=0x1ae7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.809244:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.809247:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28213248 00000020:00010000:0.0:1789760791.809254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.809258:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.809261:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.809264:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.809267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.809275:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.809281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.809298:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760791.809443:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 28209152<>28213248 00000100:00100000:0.0:1789760791.809478:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.809498:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867648:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760791.858327:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974206 gen 3 00000008:00010000:0.0:1789760791.858378:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760791.858386:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.858435:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.858521:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760791.858552:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493867648:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760791.859988:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6887, 6887] 00000080:00200000:1.0:1789760791.860021:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.860029:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.860048:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28213248 i_blocks 55305 00000020:00200000:1.0:1789760791.860054:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.860068:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.860115:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28213248, count: 4096 00000080:00200000:1.0:1789760791.860123:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28213248, bytes: 4096 00000080:00200000:1.0:1789760791.860137:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.860156:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.860159:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28213248, 28217344) 0 0 00000080:00200000:1.0:1789760791.860163:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.860180:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6888, 6888] 00020000:00200000:1.0:1789760791.860188:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28213248 chunk: [28213248, 28217344) 28217344, 4096 00020000:00200000:1.0:1789760791.860193:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.860213:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6888, 6888] 00000008:00010000:1.0:1789760791.860231:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.860237:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760791.860249:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28213248, 28217344) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.860263:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28213248=0x1ae8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.860328:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.860330:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28217344 00000020:00010000:1.0:1789760791.860338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.860342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.860345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.860348:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.860351:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.860360:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.860367:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.860422:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760791.860918:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 28213248<>28217344 00000100:00100000:1.0:1789760791.860968:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.860995:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867776:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760791.904635:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974207 gen 3 00000008:00010000:1.0:1789760791.904687:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:1.0:1789760791.904696:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.904706:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.904770:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:1.0:1789760791.904798:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493867776:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.905093:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6888, 6888] 00000080:00200000:2.0:1789760791.905110:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.905173:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.905191:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28217344 i_blocks 55305 00000020:00200000:2.0:1789760791.905194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.905204:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.905238:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28217344, count: 4096 00000080:00200000:2.0:1789760791.905246:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28217344, bytes: 4096 00000080:00200000:2.0:1789760791.905255:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.905265:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.905267:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28217344, 28221440) 0 0 00000080:00200000:2.0:1789760791.905272:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.905290:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6889, 6889] 00020000:00200000:2.0:1789760791.905298:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28217344 chunk: [28217344, 28221440) 28221440, 4096 00020000:00200000:2.0:1789760791.905302:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.905321:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6889, 6889] 00000008:00010000:2.0:1789760791.905339:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.905345:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.905357:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28217344, 28221440) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.905370:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28217344=0x1ae9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.905526:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.905530:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28221440 00000020:00010000:2.0:1789760791.905539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.905543:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.905546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.905550:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.905552:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.905559:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.905566:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.905582:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760791.905688:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 28217344<>28221440 00000100:00100000:3.0:1789760791.905758:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493867904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.944118:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974208 gen 3 00000008:00010000:3.0:1789760791.944185:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760791.944194:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.944208:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.944229:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760791.944260:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493867904:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.944640:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6889, 6889] 00000080:00200000:2.0:1789760791.944657:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.944663:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.944678:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28221440 i_blocks 55305 00000020:00200000:2.0:1789760791.944682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.944694:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.944728:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28221440, count: 4096 00000080:00200000:2.0:1789760791.944736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28221440, bytes: 4096 00000080:00200000:2.0:1789760791.944746:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.944755:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.944757:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28221440, 28225536) 0 0 00000080:00200000:2.0:1789760791.944762:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.944777:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6890, 6890] 00020000:00200000:2.0:1789760791.944786:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28221440 chunk: [28221440, 28225536) 28225536, 4096 00020000:00200000:2.0:1789760791.944790:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.944809:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6890, 6890] 00000008:00010000:2.0:1789760791.944827:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.944833:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.944844:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28221440, 28225536) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.944858:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28221440=0x1aea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.944913:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.944916:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28225536 00000020:00010000:2.0:1789760791.945692:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.945698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.945701:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.945705:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.945709:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.945718:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.945726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.945750:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760791.945863:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 28221440<>28225536 00000100:00100000:3.0:1789760791.946166:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760791.946205:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760791.985635:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974209 gen 3 00000008:00010000:3.0:1789760791.985698:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760791.985707:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.985719:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.985740:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760791.985769:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760791.986449:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6890, 6890] 00000080:00200000:2.0:1789760791.986467:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.986473:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.986489:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28225536 i_blocks 55305 00000020:00200000:2.0:1789760791.986494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.986505:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.986539:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28225536, count: 4096 00000080:00200000:2.0:1789760791.986547:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28225536, bytes: 4096 00000080:00200000:2.0:1789760791.986559:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.986568:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.986571:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28225536, 28229632) 0 0 00000080:00200000:2.0:1789760791.986575:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.986590:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6891, 6891] 00020000:00200000:2.0:1789760791.986598:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28225536 chunk: [28225536, 28229632) 28229632, 4096 00020000:00200000:2.0:1789760791.986602:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.986621:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6891, 6891] 00000008:00010000:2.0:1789760791.986639:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.986646:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760791.986659:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28225536, 28229632) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.986673:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28225536=0x1aeb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.986749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.986752:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28229632 00000020:00010000:2.0:1789760791.986760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.986763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.986766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.986770:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.986773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.986781:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.986787:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.986810:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760791.987801:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 28225536<>28229632 00000100:00100000:3.0:1789760791.988635:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.028454:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974210 gen 3 00000008:00010000:3.0:1789760792.028511:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760792.028521:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.028534:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.028553:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760792.028581:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868160:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.028722:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6891, 6891] 00000080:00200000:2.0:1789760792.028738:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.028744:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.028760:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28229632 i_blocks 55305 00000020:00200000:2.0:1789760792.028763:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.028773:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.028805:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28229632, count: 4096 00000080:00200000:2.0:1789760792.028813:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28229632, bytes: 4096 00000080:00200000:2.0:1789760792.029084:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.029095:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.029097:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28229632, 28233728) 0 0 00000080:00200000:2.0:1789760792.029102:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.029120:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6892, 6892] 00020000:00200000:2.0:1789760792.029128:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28229632 chunk: [28229632, 28233728) 28233728, 4096 00020000:00200000:2.0:1789760792.029133:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.029233:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6892, 6892] 00000008:00010000:2.0:1789760792.029282:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.029289:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.029301:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28229632, 28233728) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.029509:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28229632=0x1aec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.029573:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.029576:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28233728 00000020:00010000:2.0:1789760792.029587:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.029591:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.029594:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.029597:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.029600:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.029611:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.029618:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.029637:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760792.029745:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 28229632<>28233728 00000100:00100000:3.0:1789760792.030712:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.030744:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868288:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.083050:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974211 gen 3 00000008:00010000:3.0:1789760792.083105:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760792.083123:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.083137:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.083154:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760792.083182:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868288:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.083379:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6892, 6892] 00000080:00200000:2.0:1789760792.083424:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.083492:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.083510:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28233728 i_blocks 55305 00000020:00200000:2.0:1789760792.083514:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.083525:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.083560:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28233728, count: 4096 00000080:00200000:2.0:1789760792.083568:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28233728, bytes: 4096 00000080:00200000:2.0:1789760792.083577:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.083587:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.083589:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28233728, 28237824) 0 0 00000080:00200000:2.0:1789760792.083594:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.083610:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6893, 6893] 00020000:00200000:2.0:1789760792.083619:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28233728 chunk: [28233728, 28237824) 28237824, 4096 00020000:00200000:2.0:1789760792.083623:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.083642:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6893, 6893] 00000008:00010000:2.0:1789760792.083660:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.083666:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.083678:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28233728, 28237824) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.083691:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28233728=0x1aed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.083741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.083744:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28237824 00000020:00010000:2.0:1789760792.083752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.083755:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.083758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.083762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.083765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.083771:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.083777:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.083790:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760792.083888:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 28233728<>28237824 00000100:00100000:3.0:1789760792.084426:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868416:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.119228:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974212 gen 3 00000008:00010000:3.0:1789760792.119281:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760792.119290:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.119300:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.119430:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760792.119466:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868416:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.120149:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6893, 6893] 00000080:00200000:2.0:1789760792.120167:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.120173:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.120190:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28237824 i_blocks 55305 00000020:00200000:2.0:1789760792.120194:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.120205:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.120240:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28237824, count: 4096 00000080:00200000:2.0:1789760792.120247:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28237824, bytes: 4096 00000080:00200000:2.0:1789760792.120258:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.120267:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.120269:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28237824, 28241920) 0 0 00000080:00200000:2.0:1789760792.120273:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.120288:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6894, 6894] 00020000:00200000:2.0:1789760792.120296:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28237824 chunk: [28237824, 28241920) 28241920, 4096 00020000:00200000:2.0:1789760792.120300:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.120383:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6894, 6894] 00000008:00010000:2.0:1789760792.120429:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.120444:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.120457:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28237824, 28241920) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.120472:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28237824=0x1aee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.120528:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.120530:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28241920 00000020:00010000:2.0:1789760792.120539:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.120542:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.120546:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.120549:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.120551:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.120559:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.120566:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.120581:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760792.120687:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 28237824<>28241920 00000100:00100000:3.0:1789760792.120758:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493868544:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.162074:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974213 gen 3 00000008:00010000:3.0:1789760792.162122:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760792.162130:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.162142:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.162311:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760792.162341:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493868544:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760792.166561:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6894, 6894] 00000080:00200000:1.0:1789760792.166588:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.166595:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.166615:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28241920 i_blocks 55305 00000020:00200000:1.0:1789760792.166620:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.166640:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.166674:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28241920, count: 4096 00000080:00200000:1.0:1789760792.166682:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28241920, bytes: 4096 00000080:00200000:1.0:1789760792.166696:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.166707:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.166709:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28241920, 28246016) 0 0 00000080:00200000:1.0:1789760792.166714:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.166729:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6895, 6895] 00020000:00200000:1.0:1789760792.166737:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28241920 chunk: [28241920, 28246016) 28246016, 4096 00020000:00200000:1.0:1789760792.166742:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.166762:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6895, 6895] 00000008:00010000:1.0:1789760792.166779:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.166785:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760792.166797:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28241920, 28246016) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.166810:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28241920=0x1aef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.166871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.166874:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28246016 00000020:00010000:1.0:1789760792.166882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.166886:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.166890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.166894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.166897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.166908:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.166915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.168080:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92600 00000008:00100000:1.0:1789760792.168207:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e300 - object 0x240000400:5 offset 28241920<>28246016 00000100:00100000:0.0:1789760792.168270:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.172013:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493868672:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760792.211989:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92600 00000020:00200000:0.0:1789760792.212005:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.212017:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.212043:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92600 00000100:00100000:0.0:1789760792.212074:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493868672:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760792.212168:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6895, 6895] 00000080:00200000:3.0:1789760792.212185:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.212190:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.212205:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28246016 i_blocks 55305 00000020:00200000:3.0:1789760792.212209:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.212220:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.212253:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28246016, count: 4096 00000080:00200000:3.0:1789760792.212260:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28246016, bytes: 4096 00000080:00200000:3.0:1789760792.212270:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.212279:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.212281:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28246016, 28250112) 0 0 00000080:00200000:3.0:1789760792.212286:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.212301:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6896, 6896] 00020000:00200000:3.0:1789760792.212309:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28246016 chunk: [28246016, 28250112) 28250112, 4096 00020000:00200000:3.0:1789760792.212313:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.212604:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6896, 6896] 00000008:00010000:3.0:1789760792.212623:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.212629:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760792.212642:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28246016, 28250112) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.212655:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28246016=0x1af0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.212713:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.212716:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28250112 00000020:00010000:3.0:1789760792.212725:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.212729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.212732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.212735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.212738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.212748:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.212755:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.212773:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760792.212877:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 28246016<>28250112 00000100:00100000:2.0:1789760792.212963:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868800:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.266589:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974214 gen 3 00000100:00100000:3.0:1789760792.266634:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514b800 x1876699493868800/t4294974215(4294974215) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760808 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760792.266684:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760792.266691:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.266702:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.266782:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760792.266806:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493868800:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.266924:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6896, 6896] 00000080:00200000:0.0:1789760792.266941:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.266947:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.266970:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28250112 i_blocks 55305 00000020:00200000:0.0:1789760792.266974:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.266984:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.267088:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28250112, count: 4096 00000080:00200000:0.0:1789760792.267096:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28250112, bytes: 4096 00000080:00200000:0.0:1789760792.267108:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.267116:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.267118:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28250112, 28254208) 0 0 00000080:00200000:0.0:1789760792.267122:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.267138:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6897, 6897] 00020000:00200000:0.0:1789760792.267146:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28250112 chunk: [28250112, 28254208) 28254208, 4096 00020000:00200000:0.0:1789760792.267150:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.267168:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6897, 6897] 00000008:00010000:0.0:1789760792.267184:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.267190:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.267202:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28250112, 28254208) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.267215:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28250112=0x1af1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.267270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.267275:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28254208 00000020:00010000:0.0:1789760792.267282:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.267286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.267289:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.267292:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.267294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.267301:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.267309:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.267325:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760792.267613:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 28250112<>28254208 00000100:00100000:1.0:1789760792.267665:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760792.267779:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493868928:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760792.303096:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974216 gen 3 00000008:00010000:1.0:1789760792.303155:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760792.303163:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.303173:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.303193:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760792.303220:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493868928:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760792.303311:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6897, 6897] 00000080:00200000:3.0:1789760792.303327:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.303333:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.303349:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28254208 i_blocks 55305 00000020:00200000:3.0:1789760792.303353:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.303364:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.303428:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28254208, count: 4096 00000080:00200000:3.0:1789760792.303505:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28254208, bytes: 4096 00000080:00200000:3.0:1789760792.303517:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.303527:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.303529:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28254208, 28258304) 0 0 00000080:00200000:3.0:1789760792.303534:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.303552:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6898, 6898] 00020000:00200000:3.0:1789760792.303560:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28254208 chunk: [28254208, 28258304) 28258304, 4096 00020000:00200000:3.0:1789760792.303565:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.303585:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6898, 6898] 00000008:00010000:3.0:1789760792.303604:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.303610:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760792.303622:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28254208, 28258304) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.303638:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28254208=0x1af2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.303696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.303698:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28258304 00000020:00010000:3.0:1789760792.303706:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.303710:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.303713:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.303716:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.303719:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.303727:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.303734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.303750:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760792.303927:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 28254208<>28258304 00000100:00100000:3.0:1789760792.303986:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.304004:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493869056:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760792.345616:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760792.345629:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.345638:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.345661:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760792.345691:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493869056:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760792.345906:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6898, 6898] 00000080:00200000:1.0:1789760792.345928:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.345934:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.345971:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28258304 i_blocks 55305 00000020:00200000:1.0:1789760792.345979:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.345992:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.346036:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28258304, count: 4096 00000080:00200000:1.0:1789760792.346044:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28258304, bytes: 4096 00000080:00200000:1.0:1789760792.346058:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.346067:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.346069:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28258304, 28262400) 0 0 00000080:00200000:1.0:1789760792.346073:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.346090:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6899, 6899] 00020000:00200000:1.0:1789760792.346098:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28258304 chunk: [28258304, 28262400) 28262400, 4096 00020000:00200000:1.0:1789760792.346108:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.346128:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6899, 6899] 00000008:00010000:1.0:1789760792.346147:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.346152:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760792.346163:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28258304, 28262400) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.346177:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28258304=0x1af3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.346248:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.346252:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28262400 00000020:00010000:1.0:1789760792.346260:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.346263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.346267:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.346270:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.346274:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.346286:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.346293:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.346317:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92240 00000008:00100000:1.0:1789760792.347581:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016fb80 - object 0x240000400:5 offset 28258304<>28262400 00000100:00100000:0.0:1789760792.347703:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493869184:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760792.396957:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974218 gen 3 00000008:00010000:0.0:1789760792.397026:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92240 00000020:00200000:0.0:1789760792.397035:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.397046:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.397193:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92240 00000100:00100000:0.0:1789760792.397227:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493869184:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760792.397350:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6899, 6899] 00000080:00200000:1.0:1789760792.397368:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.397374:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.397391:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28262400 i_blocks 55305 00000020:00200000:1.0:1789760792.397425:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.397435:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.397470:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28262400, count: 4096 00000080:00200000:1.0:1789760792.397477:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28262400, bytes: 4096 00000080:00200000:1.0:1789760792.397488:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.397497:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.397500:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28262400, 28266496) 0 0 00000080:00200000:1.0:1789760792.397505:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.397522:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6900, 6900] 00020000:00200000:1.0:1789760792.397530:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28262400 chunk: [28262400, 28266496) 28266496, 4096 00020000:00200000:1.0:1789760792.397536:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.397555:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6900, 6900] 00000008:00010000:1.0:1789760792.397572:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.397579:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760792.397590:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28262400, 28266496) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.397603:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28262400=0x1af4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.397656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.397658:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28266496 00000020:00010000:1.0:1789760792.397667:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.397671:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.397674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.397678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.397681:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.397693:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.397699:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.397714:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92900 00000008:00100000:1.0:1789760792.397816:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d500 - object 0x240000400:5 offset 28262400<>28266496 00000100:00100000:0.0:1789760792.397922:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493869312:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760792.435973:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974219 gen 3 00000008:00010000:0.0:1789760792.437233:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92900 00000020:00200000:0.0:1789760792.437243:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.437254:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.437339:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92900 00000100:00100000:0.0:1789760792.437368:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493869312:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760792.437815:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6900, 6900] 00000080:00200000:1.0:1789760792.437909:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.437915:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.437931:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28266496 i_blocks 55305 00000020:00200000:1.0:1789760792.437935:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.437960:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.438174:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28266496, count: 4096 00000080:00200000:1.0:1789760792.438184:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28266496, bytes: 4096 00000080:00200000:1.0:1789760792.438199:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.438207:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.438210:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28266496, 28270592) 0 0 00000080:00200000:1.0:1789760792.438215:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.438234:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6901, 6901] 00020000:00200000:1.0:1789760792.438244:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28266496 chunk: [28266496, 28270592) 28270592, 4096 00020000:00200000:1.0:1789760792.438249:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.438269:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6901, 6901] 00000008:00010000:1.0:1789760792.438288:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.438293:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760792.438305:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28266496, 28270592) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.438321:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28266496=0x1af5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.438382:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.438385:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28270592 00000020:00010000:1.0:1789760792.438646:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.438650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.438654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.438658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.438661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.438674:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.438682:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.438704:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92000 00000008:00100000:1.0:1789760792.438814:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016f800 - object 0x240000400:5 offset 28266496<>28270592 00000100:00100000:0.0:1789760792.438898:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493869440:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760792.488808:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974220 gen 3 00000008:00010000:0.0:1789760792.488853:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92000 00000020:00200000:0.0:1789760792.488862:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.488872:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.488888:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92000 00000080:00200000:2.0:1789760792.488915:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6901, 6901] 00000100:00100000:0.0:1789760792.488918:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493869440:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.488932:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.488944:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.488966:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28270592 i_blocks 55305 00000020:00200000:2.0:1789760792.488973:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.488985:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.489027:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28270592, count: 4096 00000080:00200000:2.0:1789760792.489034:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28270592, bytes: 4096 00000080:00200000:2.0:1789760792.490241:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.490256:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.490258:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28270592, 28274688) 0 0 00000080:00200000:2.0:1789760792.490265:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.490283:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6902, 6902] 00020000:00200000:2.0:1789760792.490292:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28270592 chunk: [28270592, 28274688) 28274688, 4096 00020000:00200000:2.0:1789760792.490297:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.490319:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6902, 6902] 00000008:00010000:2.0:1789760792.490338:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.490344:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.490356:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28270592, 28274688) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.490371:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28270592=0x1af6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.490451:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.490454:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28274688 00000020:00010000:2.0:1789760792.490463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.490466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.490470:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.490473:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.490476:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.490486:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.490494:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.490517:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760792.490619:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 28270592<>28274688 00000100:00100000:2.0:1789760792.490670:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.490693:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493869568:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760792.531675:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760792.531692:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.531706:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.531772:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760792.531802:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493869568:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.531872:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6902, 6902] 00000080:00200000:2.0:1789760792.531889:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.531894:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.531909:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28274688 i_blocks 55305 00000020:00200000:2.0:1789760792.531912:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.531923:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.531968:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28274688, count: 4096 00000080:00200000:2.0:1789760792.531974:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28274688, bytes: 4096 00000080:00200000:2.0:1789760792.531985:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.531993:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.531995:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28274688, 28278784) 0 0 00000080:00200000:2.0:1789760792.531999:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.532013:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6903, 6903] 00020000:00200000:2.0:1789760792.532021:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28274688 chunk: [28274688, 28278784) 28278784, 4096 00020000:00200000:2.0:1789760792.532026:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.532043:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6903, 6903] 00000008:00010000:2.0:1789760792.532059:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.532065:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.532076:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28274688, 28278784) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.532089:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28274688=0x1af7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.532147:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.532149:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28278784 00000020:00010000:2.0:1789760792.532158:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.532162:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.532165:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.532169:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.532172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.532232:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.532242:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.532257:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760792.532347:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 28274688<>28278784 00000100:00100000:3.0:1789760792.532415:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493869696:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.572362:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974222 gen 3 00000008:00010000:3.0:1789760792.572455:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760792.572562:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.572575:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.572601:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760792.572636:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493869696:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760792.572823:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6903, 6903] 00000080:00200000:2.0:1789760792.572840:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.572845:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.572862:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28278784 i_blocks 55305 00000020:00200000:2.0:1789760792.572866:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.572877:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.572911:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28278784, count: 4096 00000080:00200000:2.0:1789760792.572918:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28278784, bytes: 4096 00000080:00200000:2.0:1789760792.572932:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.572941:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.572943:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28278784, 28282880) 0 0 00000080:00200000:2.0:1789760792.572947:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.572975:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6904, 6904] 00020000:00200000:2.0:1789760792.572983:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28278784 chunk: [28278784, 28282880) 28282880, 4096 00020000:00200000:2.0:1789760792.572987:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.573007:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6904, 6904] 00000008:00010000:2.0:1789760792.573024:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.573030:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760792.573040:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28278784, 28282880) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.573888:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28278784=0x1af8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.573970:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.573974:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28282880 00000020:00010000:2.0:1789760792.573985:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.573990:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.573993:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.573998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.574002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.574011:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.574018:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.574037:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760792.574666:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28278784<>28282880 00000100:00100000:2.0:1789760792.574729:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493869824:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.615619:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974223 gen 3 00000008:00010000:3.0:1789760792.615674:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760792.615683:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.615695:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.615715:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000080:00200000:0.0:1789760792.615742:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6904, 6904] 00000100:00100000:3.0:1789760792.615746:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493869824:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.615947:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.615964:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.615980:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28282880 i_blocks 55305 00000020:00200000:0.0:1789760792.615984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.615994:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.616026:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28282880, count: 4096 00000080:00200000:0.0:1789760792.616034:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28282880, bytes: 4096 00000080:00200000:0.0:1789760792.616044:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.616053:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.616055:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28282880, 28286976) 0 0 00000080:00200000:0.0:1789760792.616060:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.616076:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6905, 6905] 00020000:00200000:0.0:1789760792.616084:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28282880 chunk: [28282880, 28286976) 28286976, 4096 00020000:00200000:0.0:1789760792.616088:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.616107:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6905, 6905] 00000008:00010000:0.0:1789760792.616125:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.616132:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.616144:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28282880, 28286976) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.616158:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28282880=0x1af9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.616313:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.616317:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28286976 00000020:00010000:0.0:1789760792.616325:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.616329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.616332:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.616336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.616338:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.616346:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.616354:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.616370:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760792.616602:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 28282880<>28286976 00000100:00100000:1.0:1789760792.616683:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493869952:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760792.658965:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974224 gen 3 00000008:00010000:1.0:1789760792.659016:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760792.659025:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.659037:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.659057:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760792.659085:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493869952:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760792.659331:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6905, 6905] 00000080:00200000:3.0:1789760792.659347:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.659353:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.659368:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28286976 i_blocks 55305 00000020:00200000:3.0:1789760792.659372:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.659383:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.659452:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28286976, count: 4096 00000080:00200000:3.0:1789760792.659460:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28286976, bytes: 4096 00000080:00200000:3.0:1789760792.659469:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.659478:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.659480:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28286976, 28291072) 0 0 00000080:00200000:3.0:1789760792.659485:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.659501:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6906, 6906] 00020000:00200000:3.0:1789760792.659510:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28286976 chunk: [28286976, 28291072) 28291072, 4096 00020000:00200000:3.0:1789760792.659514:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.659534:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6906, 6906] 00000008:00010000:3.0:1789760792.659552:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.659558:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760792.659570:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28286976, 28291072) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.659584:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28286976=0x1afa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.659645:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.659648:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28291072 00000020:00010000:3.0:1789760792.659656:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.659659:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.659674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.659678:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.659680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.659687:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.659694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.659708:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760792.659807:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28286976<>28291072 00000100:00100000:3.0:1789760792.659854:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.659870:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493870080:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.698523:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974225 gen 3 00000008:00010000:3.0:1789760792.698579:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760792.698587:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.698598:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.698621:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760792.698647:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493870080:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.698742:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6906, 6906] 00000080:00200000:0.0:1789760792.698758:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.698764:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.698778:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28291072 i_blocks 55305 00000020:00200000:0.0:1789760792.698782:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.698791:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.698825:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28291072, count: 4096 00000080:00200000:0.0:1789760792.698831:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28291072, bytes: 4096 00000080:00200000:0.0:1789760792.698840:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.698849:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.698851:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28291072, 28295168) 0 0 00000080:00200000:0.0:1789760792.698856:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.698872:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6907, 6907] 00020000:00200000:0.0:1789760792.698880:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28291072 chunk: [28291072, 28295168) 28295168, 4096 00020000:00200000:0.0:1789760792.698885:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.698902:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6907, 6907] 00000008:00010000:0.0:1789760792.698918:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.698923:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.698933:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28291072, 28295168) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.698945:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28291072=0x1afb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.699006:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.699009:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28295168 00000020:00010000:0.0:1789760792.699017:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.699021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.699024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.699028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.699031:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.699037:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.699043:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.699057:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760792.699144:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 28291072<>28295168 00000100:00100000:1.0:1789760792.699234:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.699251:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870208:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760792.741653:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974226 gen 3 00000008:00010000:1.0:1789760792.741704:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760792.741713:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.741725:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.741747:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760792.741775:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870208:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.741893:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6907, 6907] 00000080:00200000:0.0:1789760792.741910:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.741915:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.741931:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28295168 i_blocks 55305 00000020:00200000:0.0:1789760792.741936:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.741955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.741990:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28295168, count: 4096 00000080:00200000:0.0:1789760792.741997:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28295168, bytes: 4096 00000080:00200000:0.0:1789760792.742007:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.742017:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.742019:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28295168, 28299264) 0 0 00000080:00200000:0.0:1789760792.742024:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.742040:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6908, 6908] 00020000:00200000:0.0:1789760792.742049:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28295168 chunk: [28295168, 28299264) 28299264, 4096 00020000:00200000:0.0:1789760792.742053:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.742071:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6908, 6908] 00000008:00010000:0.0:1789760792.742089:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.742095:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.742106:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28295168, 28299264) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.742119:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28295168=0x1afc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.742186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.742189:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28299264 00000020:00010000:0.0:1789760792.742197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.742200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.742203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.742207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.742210:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.742217:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.742225:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.742241:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760792.742341:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 28295168<>28299264 00000100:00100000:1.0:1789760792.742570:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870336:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760792.793890:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974227 gen 3 00000008:00010000:1.0:1789760792.794035:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760792.794044:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.794055:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.794077:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760792.794105:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870336:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.794605:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6908, 6908] 00000080:00200000:0.0:1789760792.794623:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.794629:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.794645:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28299264 i_blocks 55305 00000020:00200000:0.0:1789760792.794650:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.794661:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.794695:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28299264, count: 4096 00000080:00200000:0.0:1789760792.794702:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28299264, bytes: 4096 00000080:00200000:0.0:1789760792.794713:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.794723:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.794725:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28299264, 28303360) 0 0 00000080:00200000:0.0:1789760792.794730:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.794746:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6909, 6909] 00020000:00200000:0.0:1789760792.794755:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28299264 chunk: [28299264, 28303360) 28303360, 4096 00020000:00200000:0.0:1789760792.794759:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.794778:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6909, 6909] 00000008:00010000:0.0:1789760792.794794:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.794800:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.794811:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28299264, 28303360) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.794823:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28299264=0x1afd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.794879:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.794882:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28303360 00000020:00010000:0.0:1789760792.794890:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.794893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.794896:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.794899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.794902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.794909:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.794915:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.794941:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760792.795056:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 28299264<>28303360 00000100:00100000:1.0:1789760792.795214:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.795235:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870464:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760792.812700:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760792.826684:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974228 gen 3 00000008:00010000:1.0:1789760792.826729:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760792.826735:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.826743:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.826758:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760792.826785:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493870464:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.826786:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6909, 6909] 00000080:00200000:0.0:1789760792.826803:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.826808:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.826823:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28303360 i_blocks 55305 00000020:00200000:0.0:1789760792.826827:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.826837:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.826873:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28303360, count: 4096 00000080:00200000:0.0:1789760792.826881:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28303360, bytes: 4096 00000080:00200000:0.0:1789760792.826889:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.826928:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.826931:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28303360, 28307456) 0 0 00000080:00200000:0.0:1789760792.826936:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.826962:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6910, 6910] 00020000:00200000:0.0:1789760792.826971:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28303360 chunk: [28303360, 28307456) 28307456, 4096 00020000:00200000:0.0:1789760792.826976:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.826998:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6910, 6910] 00000008:00010000:0.0:1789760792.827016:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.827022:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.827033:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28303360, 28307456) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.827046:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28303360=0x1afe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.827113:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.827116:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28307456 00000020:00010000:0.0:1789760792.827124:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.827128:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.827131:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.827136:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.827138:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.827146:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.827152:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.827168:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760792.827414:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 28303360<>28307456 00000100:00100000:0.0:1789760792.827457:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760792.827485:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493870592:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760792.870079:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:0.0:1789760792.870094:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.870105:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.870131:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:0.0:1789760792.870160:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493870592:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760792.870255:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6910, 6910] 00000080:00200000:3.0:1789760792.870272:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.870277:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.870293:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28307456 i_blocks 55305 00000020:00200000:3.0:1789760792.870297:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.870307:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.870342:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28307456, count: 4096 00000080:00200000:3.0:1789760792.870349:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28307456, bytes: 4096 00000080:00200000:3.0:1789760792.870358:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.870366:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.870369:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28307456, 28311552) 0 0 00000080:00200000:3.0:1789760792.870373:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.870388:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6911, 6911] 00020000:00200000:3.0:1789760792.870422:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28307456 chunk: [28307456, 28311552) 28311552, 4096 00020000:00200000:3.0:1789760792.870427:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.870446:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6911, 6911] 00000008:00010000:3.0:1789760792.870464:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.870470:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760792.870482:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28307456, 28311552) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.870496:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28307456=0x1aff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.870555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.870558:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28311552 00000020:00010000:3.0:1789760792.870565:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.870568:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.870572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.870575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.870578:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.870584:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.870590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.870606:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760792.870704:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28307456<>28311552 00000100:00100000:2.0:1789760792.870779:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493870720:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760792.914664:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974230 gen 3 00000008:00010000:3.0:1789760792.914714:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760792.914721:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.914728:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.914808:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760792.914834:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493870720:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760792.914903:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6911, 6911] 00000080:00200000:0.0:1789760792.914915:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.914920:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.914931:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28311552 i_blocks 55305 00000020:00200000:0.0:1789760792.914982:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.914991:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.915019:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28311552, count: 4096 00000080:00200000:0.0:1789760792.915026:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28311552, bytes: 4096 00000080:00200000:0.0:1789760792.915035:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.915041:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.915044:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28311552, 28315648) 0 0 00000080:00200000:0.0:1789760792.915048:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.915060:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6912, 6912] 00020000:00200000:0.0:1789760792.915067:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28311552 chunk: [28311552, 28315648) 28315648, 4096 00020000:00200000:0.0:1789760792.915071:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.915086:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6912, 6912] 00000008:00010000:0.0:1789760792.915099:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.915105:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760792.915114:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28311552, 28315648) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.915124:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28311552=0x1b00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.915168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.915171:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28315648 00000020:00010000:0.0:1789760792.915178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.915181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.915184:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.915187:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.915190:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.915196:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.915201:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.915217:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760792.915290:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ea00 - object 0x240000400:5 offset 28311552<>28315648 00000100:00100000:1.0:1789760792.915371:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493870848:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760792.962341:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974231 gen 3 00000008:00010000:1.0:1789760792.962383:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760792.962391:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.962433:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.962459:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760792.962495:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493870848:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760792.962835:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6912, 6912] 00000080:00200000:3.0:1789760792.962853:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.962859:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.962874:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28315648 i_blocks 55305 00000020:00200000:3.0:1789760792.962878:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.962890:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.962929:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28315648, count: 4096 00000080:00200000:3.0:1789760792.962935:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28315648, bytes: 4096 00000080:00200000:3.0:1789760792.963047:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.963056:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.963058:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28315648, 28319744) 0 0 00000080:00200000:3.0:1789760792.963063:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.963076:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6913, 6913] 00020000:00200000:3.0:1789760792.963083:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28315648 chunk: [28315648, 28319744) 28319744, 4096 00020000:00200000:3.0:1789760792.963087:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.963104:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6913, 6913] 00000008:00010000:3.0:1789760792.963119:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.963124:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760792.963133:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28315648, 28319744) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.963144:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28315648=0x1b01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.963191:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.963193:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28319744 00000020:00010000:3.0:1789760792.963200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.963203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.963206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.963208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.963211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.963216:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.963222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.963235:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760792.963314:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28315648<>28319744 00000100:00100000:2.0:1789760792.963381:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493870976:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.014467:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974232 gen 3 00000008:00010000:3.0:1789760793.014502:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760793.014509:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.014517:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.014537:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760793.014562:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493870976:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760793.014845:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6913, 6913] 00000080:00200000:0.0:1789760793.014856:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.014861:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.014871:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28319744 i_blocks 57353 00000020:00200000:0.0:1789760793.014874:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.014882:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.014904:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28319744, count: 4096 00000080:00200000:0.0:1789760793.014910:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28319744, bytes: 4096 00000080:00200000:0.0:1789760793.014918:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.014925:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.014927:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28319744, 28323840) 0 0 00000080:00200000:0.0:1789760793.014930:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.014942:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6914, 6914] 00020000:00200000:0.0:1789760793.014967:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28319744 chunk: [28319744, 28323840) 28323840, 4096 00020000:00200000:0.0:1789760793.014972:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.014986:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6914, 6914] 00000008:00010000:0.0:1789760793.015001:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.015006:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760793.015015:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28319744, 28323840) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.015027:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28319744=0x1b02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.015072:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.015075:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28323840 00000020:00010000:0.0:1789760793.015081:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.015084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.015087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.015090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.015093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.015099:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.015104:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.015119:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760793.015197:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304df80 - object 0x240000400:5 offset 28319744<>28323840 00000100:00100000:1.0:1789760793.015624:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493871104:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760793.051113:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760793.051124:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.051134:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.051537:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760793.051564:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493871104:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760793.051653:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6914, 6914] 00000080:00200000:0.0:1789760793.051666:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.051671:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.051683:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28323840 i_blocks 57353 00000020:00200000:0.0:1789760793.051686:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.051695:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.051722:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28323840, count: 4096 00000080:00200000:0.0:1789760793.051727:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28323840, bytes: 4096 00000080:00200000:0.0:1789760793.051738:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.051745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.051747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28323840, 28327936) 0 0 00000080:00200000:0.0:1789760793.051751:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.051765:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6915, 6915] 00020000:00200000:0.0:1789760793.051772:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28323840 chunk: [28323840, 28327936) 28327936, 4096 00020000:00200000:0.0:1789760793.051776:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.051792:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6915, 6915] 00000008:00010000:0.0:1789760793.051805:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.051810:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760793.051819:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28323840, 28327936) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.051829:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28323840=0x1b03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.051872:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.051875:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28327936 00000020:00010000:0.0:1789760793.051882:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.051885:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.051888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.051891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.051895:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.051901:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.051906:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.051921:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760793.052109:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ca80 - object 0x240000400:5 offset 28323840<>28327936 00000100:00100000:1.0:1789760793.052169:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493871232:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760793.090982:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974233 gen 3 00000100:00100000:1.0:1789760793.091031:0:2368:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304ca80 x1876699493871232/t4294974234(4294974234) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760793.091089:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760793.091097:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.091106:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.091169:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760793.091193:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493871232:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.091359:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6915, 6915] 00000080:00200000:2.0:1789760793.091376:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.091382:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.091490:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28327936 i_blocks 57353 00000020:00200000:2.0:1789760793.091495:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.091522:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.091559:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28327936, count: 4096 00000080:00200000:2.0:1789760793.091567:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28327936, bytes: 4096 00000080:00200000:2.0:1789760793.091577:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.091587:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.091589:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28327936, 28332032) 0 0 00000080:00200000:2.0:1789760793.091594:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.091611:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6916, 6916] 00020000:00200000:2.0:1789760793.091619:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28327936 chunk: [28327936, 28332032) 28332032, 4096 00020000:00200000:2.0:1789760793.091624:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.091643:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6916, 6916] 00000008:00010000:2.0:1789760793.091661:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.091667:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.091679:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28327936, 28332032) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.091693:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28327936=0x1b04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.091747:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.091750:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28332032 00000020:00010000:2.0:1789760793.091758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.091762:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.091765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.091769:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.091772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.091779:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.091785:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.091796:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760793.091896:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 28327936<>28332032 00000100:00100000:3.0:1789760793.091975:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493871360:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760793.135007:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493871488:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.135143:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493871744:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.135186:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760793.135192:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493871616:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.141602:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493871616:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.141625:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2367:1876699493871488:192.168.203.160@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.141638:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2367:1876699493871744:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760793.171287:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974234 gen 3 00000100:00100000:3.0:1789760793.171326:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83480 x1876699493871360/t4294974235(4294974235) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.171374:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760793.171382:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.171420:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.171439:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760793.171462:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493871360:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.171628:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6916, 6916] 00000080:00200000:2.0:1789760793.171646:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.171652:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.171668:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28332032 i_blocks 57353 00000020:00200000:2.0:1789760793.171673:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.171682:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.171718:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28332032, count: 4096 00000080:00200000:2.0:1789760793.171726:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28332032, bytes: 4096 00000080:00200000:2.0:1789760793.171735:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.171745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.171747:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28332032, 28336128) 0 0 00000080:00200000:2.0:1789760793.171751:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.171768:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6917, 6917] 00020000:00200000:2.0:1789760793.171776:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28332032 chunk: [28332032, 28336128) 28336128, 4096 00020000:00200000:2.0:1789760793.171781:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.171800:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6917, 6917] 00000008:00010000:2.0:1789760793.171817:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.171823:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.171835:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28332032, 28336128) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.171849:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28332032=0x1b05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.171907:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.171911:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28336128 00000020:00010000:2.0:1789760793.171920:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.171924:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.171927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.171930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.171933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.171941:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.171963:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.171980:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760793.172092:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 28332032<>28336128 00000100:00100000:3.0:1789760793.172295:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.172312:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493871872:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.223987:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974235 gen 3 00000100:00100000:3.0:1789760793.224037:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f80e00 x1876699493871872/t4294974236(4294974236) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.224095:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760793.224104:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.224115:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.224134:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760793.224156:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493871872:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.224294:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6917, 6917] 00000080:00200000:2.0:1789760793.224310:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.224316:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.224331:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28336128 i_blocks 57353 00000020:00200000:2.0:1789760793.224345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.224355:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.224389:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28336128, count: 4096 00000080:00200000:2.0:1789760793.224437:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28336128, bytes: 4096 00000080:00200000:2.0:1789760793.224448:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.224457:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.224460:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28336128, 28340224) 0 0 00000080:00200000:2.0:1789760793.224464:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.224480:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6918, 6918] 00020000:00200000:2.0:1789760793.224489:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28336128 chunk: [28336128, 28340224) 28340224, 4096 00020000:00200000:2.0:1789760793.224495:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.224514:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6918, 6918] 00000008:00010000:2.0:1789760793.224531:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.224537:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.224549:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28336128, 28340224) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.224563:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28336128=0x1b06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.224619:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.224623:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28340224 00000020:00010000:2.0:1789760793.224631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.224634:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.224637:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.224641:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.224643:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.224651:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.224659:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.224674:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760793.224776:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 28336128<>28340224 00000100:00100000:3.0:1789760793.225079:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872000:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.260745:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974236 gen 3 00000100:00100000:3.0:1789760793.260790:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f82a00 x1876699493872000/t4294974237(4294974237) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.260846:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760793.260853:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.260864:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.260886:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760793.260911:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872000:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.261002:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6918, 6918] 00000080:00200000:2.0:1789760793.261020:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.261026:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.261042:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28340224 i_blocks 57353 00000020:00200000:2.0:1789760793.261046:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.261056:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.261152:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28340224, count: 4096 00000080:00200000:2.0:1789760793.261160:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28340224, bytes: 4096 00000080:00200000:2.0:1789760793.261170:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.261179:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.261182:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28340224, 28344320) 0 0 00000080:00200000:2.0:1789760793.261187:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.261205:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6919, 6919] 00020000:00200000:2.0:1789760793.261213:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28340224 chunk: [28340224, 28344320) 28344320, 4096 00020000:00200000:2.0:1789760793.261218:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.261238:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6919, 6919] 00000008:00010000:2.0:1789760793.261256:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.261263:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.261275:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28340224, 28344320) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.261289:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28340224=0x1b07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.261346:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.261350:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28344320 00000020:00010000:2.0:1789760793.261358:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.261362:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.261366:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.261369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.261373:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.261381:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.261388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.261595:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760793.261698:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 28340224<>28344320 00000100:00100000:3.0:1789760793.261911:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.261929:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872128:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.324674:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974237 gen 3 00000100:00100000:3.0:1789760793.324716:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81880 x1876699493872128/t4294974238(4294974238) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.324770:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760793.324778:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.324789:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.324868:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760793.324895:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872128:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760793.325066:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6919, 6919] 00000080:00200000:0.0:1789760793.325084:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.325090:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.325105:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28344320 i_blocks 57353 00000020:00200000:0.0:1789760793.325109:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.325120:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.325153:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28344320, count: 4096 00000080:00200000:0.0:1789760793.325160:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28344320, bytes: 4096 00000080:00200000:0.0:1789760793.325171:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.325181:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.325183:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28344320, 28348416) 0 0 00000080:00200000:0.0:1789760793.325188:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.325204:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6920, 6920] 00020000:00200000:0.0:1789760793.325213:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28344320 chunk: [28344320, 28348416) 28348416, 4096 00020000:00200000:0.0:1789760793.325218:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.325238:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6920, 6920] 00000008:00010000:0.0:1789760793.325256:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.325262:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760793.325273:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28344320, 28348416) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.325287:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28344320=0x1b08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.325348:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.325351:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28348416 00000020:00010000:0.0:1789760793.325359:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.325363:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.325366:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.325369:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.325493:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.325504:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.325510:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.325525:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760793.325611:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 28344320<>28348416 00000100:00100000:0.0:1789760793.325680:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872256:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.356142:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974239 gen 3 00000008:00010000:0.0:1789760793.356208:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760793.356216:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.356227:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.356296:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760793.356326:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872256:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760793.356540:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6920, 6920] 00000080:00200000:3.0:1789760793.356557:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.356563:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.356579:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28348416 i_blocks 57353 00000020:00200000:3.0:1789760793.356583:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.356593:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.356715:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28348416, count: 4096 00000080:00200000:3.0:1789760793.356725:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28348416, bytes: 4096 00000080:00200000:3.0:1789760793.356737:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.356746:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.356748:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28348416, 28352512) 0 0 00000080:00200000:3.0:1789760793.356753:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.356769:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6921, 6921] 00020000:00200000:3.0:1789760793.356778:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28348416 chunk: [28348416, 28352512) 28352512, 4096 00020000:00200000:3.0:1789760793.356783:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.356802:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6921, 6921] 00000008:00010000:3.0:1789760793.356822:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.356828:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760793.356900:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28348416, 28352512) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.356924:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28348416=0x1b09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.356992:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.356995:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28352512 00000020:00010000:3.0:1789760793.357004:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.357008:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.357011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.357015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.357018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.357026:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.357032:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.357049:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760793.357149:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28348416<>28352512 00000100:00100000:2.0:1789760793.357219:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872384:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.388510:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974240 gen 3 00000008:00010000:3.0:1789760793.388564:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760793.388572:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.388583:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.388605:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000080:00200000:0.0:1789760793.388635:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6921, 6921] 00000100:00100000:3.0:1789760793.388635:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493872384:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760793.388652:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.388658:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.388674:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28352512 i_blocks 57353 00000020:00200000:0.0:1789760793.388679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.388689:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.388722:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28352512, count: 4096 00000080:00200000:0.0:1789760793.388731:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28352512, bytes: 4096 00000080:00200000:0.0:1789760793.388740:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.388748:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.388750:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28352512, 28356608) 0 0 00000080:00200000:0.0:1789760793.388756:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.388771:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6922, 6922] 00020000:00200000:0.0:1789760793.388780:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28352512 chunk: [28352512, 28356608) 28356608, 4096 00020000:00200000:0.0:1789760793.388785:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.388807:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6922, 6922] 00000008:00010000:0.0:1789760793.388825:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.388831:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760793.388849:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28352512, 28356608) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.388863:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28352512=0x1b0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.388920:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.388923:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28356608 00000020:00010000:0.0:1789760793.388931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.388935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.388938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.388950:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.388954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.388961:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.388967:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.388983:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760793.389085:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 28352512<>28356608 00000100:00100000:0.0:1789760793.389193:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.389210:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872512:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.435477:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974241 gen 3 00000008:00010000:0.0:1789760793.436183:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760793.436194:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.436205:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.436391:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760793.436444:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872512:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760793.436996:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6922, 6922] 00000080:00200000:1.0:1789760793.437014:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.437020:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.437036:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28356608 i_blocks 57353 00000020:00200000:1.0:1789760793.437040:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.437052:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.437087:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28356608, count: 4096 00000080:00200000:1.0:1789760793.437094:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28356608, bytes: 4096 00000080:00200000:1.0:1789760793.437108:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.437117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.437120:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28356608, 28360704) 0 0 00000080:00200000:1.0:1789760793.437125:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.437142:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6923, 6923] 00020000:00200000:1.0:1789760793.437151:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28356608 chunk: [28356608, 28360704) 28360704, 4096 00020000:00200000:1.0:1789760793.437155:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.437176:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6923, 6923] 00000008:00010000:1.0:1789760793.437195:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.437202:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760793.437215:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28356608, 28360704) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.437298:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28356608=0x1b0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.437364:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.437367:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28360704 00000020:00010000:1.0:1789760793.437377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.437381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.437384:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.437387:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.437391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.437483:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.437490:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.437635:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92d80 00000008:00100000:1.0:1789760793.437750:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016e680 - object 0x240000400:5 offset 28356608<>28360704 00000100:00100000:0.0:1789760793.437818:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872640:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.492999:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974242 gen 3 00000008:00010000:0.0:1789760793.493053:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92d80 00000020:00200000:0.0:1789760793.493062:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.493073:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.493157:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92d80 00000100:00100000:0.0:1789760793.493204:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493872640:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760793.493297:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6923, 6923] 00000080:00200000:1.0:1789760793.493314:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.493320:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.493336:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28360704 i_blocks 57353 00000020:00200000:1.0:1789760793.493341:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.493351:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.493384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28360704, count: 4096 00000080:00200000:1.0:1789760793.493392:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28360704, bytes: 4096 00000080:00200000:1.0:1789760793.493459:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.493467:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.493470:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28360704, 28364800) 0 0 00000080:00200000:1.0:1789760793.493475:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.493493:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6924, 6924] 00020000:00200000:1.0:1789760793.493502:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28360704 chunk: [28360704, 28364800) 28364800, 4096 00020000:00200000:1.0:1789760793.493506:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.493527:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6924, 6924] 00000008:00010000:1.0:1789760793.493544:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.493550:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760793.493562:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28360704, 28364800) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.493577:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28360704=0x1b0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.493633:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.493636:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28364800 00000020:00010000:1.0:1789760793.493644:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.493648:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.493651:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.493654:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.493658:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.493670:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.493677:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.493693:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92e40 00000008:00100000:1.0:1789760793.493797:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016c380 - object 0x240000400:5 offset 28360704<>28364800 00000100:00100000:0.0:1789760793.493870:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493872768:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.534810:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974243 gen 3 00000008:00010000:0.0:1789760793.534864:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92e40 00000020:00200000:0.0:1789760793.534872:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.534883:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.534907:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92e40 00000100:00100000:0.0:1789760793.534936:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493872768:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760793.535058:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6924, 6924] 00000080:00200000:1.0:1789760793.535076:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.535081:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.535097:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28364800 i_blocks 57353 00000020:00200000:1.0:1789760793.535101:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.535111:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.535143:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28364800, count: 4096 00000080:00200000:1.0:1789760793.535150:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28364800, bytes: 4096 00000080:00200000:1.0:1789760793.535158:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.535177:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.535179:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28364800, 28368896) 0 0 00000080:00200000:1.0:1789760793.535183:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.535199:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6925, 6925] 00020000:00200000:1.0:1789760793.535206:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28364800 chunk: [28364800, 28368896) 28368896, 4096 00020000:00200000:1.0:1789760793.535211:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.535230:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6925, 6925] 00000008:00010000:1.0:1789760793.535247:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.535254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760793.535277:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28364800, 28368896) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.535291:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28364800=0x1b0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.535348:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.535350:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28368896 00000020:00010000:1.0:1789760793.535359:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.535362:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.535377:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.535381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.535385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.535418:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.535434:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.535449:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92f00 00000008:00100000:1.0:1789760793.535550:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016d880 - object 0x240000400:5 offset 28364800<>28368896 00000100:00100000:0.0:1789760793.535996:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.536018:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493872896:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.587595:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974244 gen 3 00000008:00010000:0.0:1789760793.587648:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92f00 00000020:00200000:0.0:1789760793.587657:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.587668:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.587693:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92f00 00000100:00100000:0.0:1789760793.587725:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493872896:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760793.587893:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6925, 6925] 00000080:00200000:1.0:1789760793.587908:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760793.587914:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760793.587928:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28368896 i_blocks 57353 00000020:00200000:1.0:1789760793.587931:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760793.587951:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760793.587982:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28368896, count: 4096 00000080:00200000:1.0:1789760793.587988:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28368896, bytes: 4096 00000080:00200000:1.0:1789760793.588058:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760793.588067:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760793.588069:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28368896, 28372992) 0 0 00000080:00200000:1.0:1789760793.588074:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760793.588089:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6926, 6926] 00020000:00200000:1.0:1789760793.588097:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28368896 chunk: [28368896, 28372992) 28372992, 4096 00020000:00200000:1.0:1789760793.588101:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760793.588120:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6926, 6926] 00000008:00010000:1.0:1789760793.588137:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760793.588144:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760793.588155:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28368896, 28372992) append 0 append lockpos 0 00000080:00200000:1.0:1789760793.588170:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28368896=0x1b0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760793.588230:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760793.588233:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28372992 00000020:00010000:1.0:1789760793.588241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760793.588245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.588247:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760793.588251:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760793.588254:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760793.588263:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760793.588269:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760793.588283:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e929c0 00000008:00100000:1.0:1789760793.588387:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 28368896<>28372992 00000100:00100000:0.0:1789760793.588886:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493873024:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760793.621091:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974245 gen 3 00000008:00010000:0.0:1789760793.621140:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e929c0 00000020:00200000:0.0:1789760793.621148:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.621157:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.621250:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e929c0 00000100:00100000:0.0:1789760793.621280:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493873024:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760793.621642:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6926, 6926] 00000080:00200000:3.0:1789760793.621659:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760793.621666:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760793.621682:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28372992 i_blocks 57353 00000020:00200000:3.0:1789760793.621687:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760793.621697:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760793.621729:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28372992, count: 4096 00000080:00200000:3.0:1789760793.621736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28372992, bytes: 4096 00000080:00200000:3.0:1789760793.621747:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760793.621756:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760793.621758:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28372992, 28377088) 0 0 00000080:00200000:3.0:1789760793.621763:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760793.621780:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6927, 6927] 00020000:00200000:3.0:1789760793.621789:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28372992 chunk: [28372992, 28377088) 28377088, 4096 00020000:00200000:3.0:1789760793.621793:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760793.621811:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6927, 6927] 00000008:00010000:3.0:1789760793.621828:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760793.621834:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760793.621845:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28372992, 28377088) append 0 append lockpos 0 00000080:00200000:3.0:1789760793.621860:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28372992=0x1b0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760793.621916:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760793.621919:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28377088 00000020:00010000:3.0:1789760793.621927:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760793.621930:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.621933:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760793.621937:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760793.621951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760793.621959:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760793.621966:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760793.621982:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760793.622081:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28372992<>28377088 00000100:00100000:3.0:1789760793.622129:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.622147:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873152:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.657354:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974246 gen 3 00000008:00010000:3.0:1789760793.657576:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760793.657585:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.657597:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.657616:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760793.657645:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873152:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.657751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6927, 6927] 00000080:00200000:2.0:1789760793.657767:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.657774:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.657790:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28377088 i_blocks 57353 00000020:00200000:2.0:1789760793.657793:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.657803:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.657887:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28377088, count: 4096 00000080:00200000:2.0:1789760793.657895:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28377088, bytes: 4096 00000080:00200000:2.0:1789760793.657904:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.657912:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.657914:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28377088, 28381184) 0 0 00000080:00200000:2.0:1789760793.657919:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.657933:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6928, 6928] 00020000:00200000:2.0:1789760793.657951:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28377088 chunk: [28377088, 28381184) 28381184, 4096 00020000:00200000:2.0:1789760793.657955:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.657974:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6928, 6928] 00000008:00010000:2.0:1789760793.657990:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.658224:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.658238:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28377088, 28381184) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.658253:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28377088=0x1b10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.658306:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.658309:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28381184 00000020:00010000:2.0:1789760793.658316:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.658320:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.658323:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.658326:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.658329:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.658336:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.658342:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.658355:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760793.658605:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 28377088<>28381184 00000100:00100000:3.0:1789760793.658672:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873280:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760793.698506:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760793.698523:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.698535:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.698555:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760793.698585:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873280:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.698767:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6928, 6928] 00000080:00200000:2.0:1789760793.698782:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.698788:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.698804:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28381184 i_blocks 57353 00000020:00200000:2.0:1789760793.698808:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.698818:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.698851:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28381184, count: 4096 00000080:00200000:2.0:1789760793.698858:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28381184, bytes: 4096 00000080:00200000:2.0:1789760793.698867:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.698876:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.698879:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28381184, 28385280) 0 0 00000080:00200000:2.0:1789760793.698883:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.698898:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6929, 6929] 00020000:00200000:2.0:1789760793.698906:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28381184 chunk: [28381184, 28385280) 28385280, 4096 00020000:00200000:2.0:1789760793.698911:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.698928:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6929, 6929] 00000008:00010000:2.0:1789760793.699024:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.699034:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.699047:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28381184, 28385280) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.699061:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28381184=0x1b11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.699116:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.699119:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28385280 00000020:00010000:2.0:1789760793.699125:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.699129:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.699131:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.699135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.699137:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.699145:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.699151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.699173:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760793.699250:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 28381184<>28385280 00000100:00100000:3.0:1789760793.699546:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873408:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.748454:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974247 gen 3 00000100:00100000:3.0:1789760793.748498:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81c00 x1876699493873408/t4294974248(4294974248) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.748546:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760793.748554:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.748565:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.748587:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760793.748609:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873408:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.748753:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6929, 6929] 00000080:00200000:2.0:1789760793.748770:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.748776:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.748792:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28385280 i_blocks 57353 00000020:00200000:2.0:1789760793.748796:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.748807:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.748839:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28385280, count: 4096 00000080:00200000:2.0:1789760793.748847:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28385280, bytes: 4096 00000080:00200000:2.0:1789760793.748857:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.748866:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.748868:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28385280, 28389376) 0 0 00000080:00200000:2.0:1789760793.748873:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.748889:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6930, 6930] 00020000:00200000:2.0:1789760793.748907:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28385280 chunk: [28385280, 28389376) 28389376, 4096 00020000:00200000:2.0:1789760793.748912:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.748931:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6930, 6930] 00000008:00010000:2.0:1789760793.748957:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.748965:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.748977:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28385280, 28389376) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.749001:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28385280=0x1b12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.749061:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.749065:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28389376 00000020:00010000:2.0:1789760793.749073:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.749077:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.749080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.749084:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.749087:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.749094:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.749100:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.749115:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760793.749220:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 28385280<>28389376 00000100:00100000:3.0:1789760793.749446:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873536:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.789060:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974249 gen 3 00000008:00010000:3.0:1789760793.789123:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760793.789132:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.789143:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.789219:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760793.789248:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493873536:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760793.792566:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6930, 6930] 00000080:00200000:0.0:1789760793.792593:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.792600:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.792619:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28389376 i_blocks 57353 00000020:00200000:0.0:1789760793.792625:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.792639:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.792673:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28389376, count: 4096 00000080:00200000:0.0:1789760793.792682:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28389376, bytes: 4096 00000080:00200000:0.0:1789760793.792695:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.792704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.792706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28389376, 28393472) 0 0 00000080:00200000:0.0:1789760793.792710:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.792726:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6931, 6931] 00020000:00200000:0.0:1789760793.792733:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28389376 chunk: [28389376, 28393472) 28393472, 4096 00020000:00200000:0.0:1789760793.792737:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.792756:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6931, 6931] 00000008:00010000:0.0:1789760793.792774:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.792781:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760793.792794:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28389376, 28393472) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.792808:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28389376=0x1b13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.792869:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.792873:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28393472 00000020:00010000:0.0:1789760793.792881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.792884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.792888:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.792891:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.792894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.792902:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.792909:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.792936:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760793.795837:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f100 - object 0x240000400:5 offset 28389376<>28393472 00000100:00100000:0.0:1789760793.795901:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760793.795926:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493873664:192.168.203.160@tcp:4:dd.0 00000008:00010000:0.0:1789760793.834444:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:0.0:1789760793.834461:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.834472:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.834501:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:0.0:1789760793.834530:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493873664:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.834648:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6931, 6931] 00000080:00200000:2.0:1789760793.834664:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.834669:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.834682:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28393472 i_blocks 57353 00000020:00200000:2.0:1789760793.834685:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.834695:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.834731:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28393472, count: 4096 00000080:00200000:2.0:1789760793.834738:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28393472, bytes: 4096 00000080:00200000:2.0:1789760793.834750:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.834759:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.834762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28393472, 28397568) 0 0 00000080:00200000:2.0:1789760793.834766:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.834783:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6932, 6932] 00020000:00200000:2.0:1789760793.834791:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28393472 chunk: [28393472, 28397568) 28397568, 4096 00020000:00200000:2.0:1789760793.834795:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.834815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6932, 6932] 00000008:00010000:2.0:1789760793.834834:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.834839:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.834851:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28393472, 28397568) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.834865:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28393472=0x1b14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.834923:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.834926:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28397568 00000020:00010000:2.0:1789760793.834935:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.834938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.834951:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.834955:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.834958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.834965:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.834971:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.834987:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760793.835090:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82680 - object 0x240000400:5 offset 28393472<>28397568 00000100:00100000:3.0:1789760793.835680:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873792:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760793.836696:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760793.875979:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974250 gen 3 00000100:00100000:3.0:1789760793.876033:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f82680 x1876699493873792/t4294974251(4294974251) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.876082:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760793.876090:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.876101:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.876121:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760793.876143:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873792:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.876292:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6932, 6932] 00000080:00200000:2.0:1789760793.876309:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.876314:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.876330:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28397568 i_blocks 57353 00000020:00200000:2.0:1789760793.876334:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.876345:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.876376:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28397568, count: 4096 00000080:00200000:2.0:1789760793.876383:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28397568, bytes: 4096 00000080:00200000:2.0:1789760793.876429:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.876438:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.876440:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28397568, 28401664) 0 0 00000080:00200000:2.0:1789760793.876444:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.876459:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6933, 6933] 00020000:00200000:2.0:1789760793.876467:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28397568 chunk: [28397568, 28401664) 28401664, 4096 00020000:00200000:2.0:1789760793.876472:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.876490:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6933, 6933] 00000008:00010000:2.0:1789760793.876579:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.876585:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.876597:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28397568, 28401664) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.876611:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28397568=0x1b15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.876665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.876668:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28401664 00000020:00010000:2.0:1789760793.876676:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.876680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.876683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.876686:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.876689:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.876698:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.876704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.876721:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b840 00000008:00100000:2.0:1789760793.876821:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82d80 - object 0x240000400:5 offset 28397568<>28401664 00000100:00100000:3.0:1789760793.877115:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.877135:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873920:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.914780:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974251 gen 3 00000100:00100000:3.0:1789760793.914828:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f82d80 x1876699493873920/t4294974252(4294974252) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.914877:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b840 00000020:00200000:3.0:1789760793.914885:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.914895:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.914915:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b840 00000100:00100000:3.0:1789760793.914947:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493873920:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.915131:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6933, 6933] 00000080:00200000:2.0:1789760793.915146:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.915152:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.915167:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28401664 i_blocks 57353 00000020:00200000:2.0:1789760793.915171:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.915181:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.915215:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28401664, count: 4096 00000080:00200000:2.0:1789760793.915222:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28401664, bytes: 4096 00000080:00200000:2.0:1789760793.915232:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.915240:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.915242:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28401664, 28405760) 0 0 00000080:00200000:2.0:1789760793.915247:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.915263:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6934, 6934] 00020000:00200000:2.0:1789760793.915273:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28401664 chunk: [28401664, 28405760) 28405760, 4096 00020000:00200000:2.0:1789760793.915277:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.915296:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6934, 6934] 00000008:00010000:2.0:1789760793.915312:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.915319:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.915329:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28401664, 28405760) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.915341:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28401664=0x1b16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.915446:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.915449:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28405760 00000020:00010000:2.0:1789760793.915457:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.915460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.915463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.915467:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.915469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.915477:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.915483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.915497:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000008:00100000:2.0:1789760793.915591:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81c00 - object 0x240000400:5 offset 28401664<>28405760 00000100:00100000:3.0:1789760793.915744:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874048:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.951823:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974253 gen 3 00000008:00010000:3.0:1789760793.951892:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000020:00200000:3.0:1789760793.951901:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.951913:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.951936:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b6c0 00000100:00100000:3.0:1789760793.951976:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874048:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.952138:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6934, 6934] 00000080:00200000:2.0:1789760793.952155:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.952161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.952177:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28405760 i_blocks 57353 00000020:00200000:2.0:1789760793.952181:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.952192:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.952227:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28405760, count: 4096 00000080:00200000:2.0:1789760793.952234:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28405760, bytes: 4096 00000080:00200000:2.0:1789760793.952243:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.952251:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.952253:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28405760, 28409856) 0 0 00000080:00200000:2.0:1789760793.952258:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.952275:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6935, 6935] 00020000:00200000:2.0:1789760793.952283:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28405760 chunk: [28405760, 28409856) 28409856, 4096 00020000:00200000:2.0:1789760793.952288:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.952308:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6935, 6935] 00000008:00010000:2.0:1789760793.952327:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.952334:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.952347:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28405760, 28409856) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.952361:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28405760=0x1b17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.952511:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.952515:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28409856 00000020:00010000:2.0:1789760793.952524:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.952528:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.952531:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.952534:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.952537:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.952545:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.952551:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.952565:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bc00 00000008:00100000:2.0:1789760793.952667:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80700 - object 0x240000400:5 offset 28405760<>28409856 00000100:00100000:3.0:1789760793.953007:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.953027:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874176:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760793.985229:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974254 gen 3 00000008:00010000:3.0:1789760793.985282:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bc00 00000020:00200000:3.0:1789760793.985291:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.985304:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.985368:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bc00 00000100:00100000:3.0:1789760793.985441:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874176:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760793.985534:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6935, 6935] 00000080:00200000:2.0:1789760793.985550:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.985557:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.985573:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28409856 i_blocks 57353 00000020:00200000:2.0:1789760793.985577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.985587:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.985622:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28409856, count: 4096 00000080:00200000:2.0:1789760793.985629:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28409856, bytes: 4096 00000080:00200000:2.0:1789760793.985638:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.985647:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.985649:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28409856, 28413952) 0 0 00000080:00200000:2.0:1789760793.985653:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.985670:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6936, 6936] 00020000:00200000:2.0:1789760793.985679:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28409856 chunk: [28409856, 28413952) 28413952, 4096 00020000:00200000:2.0:1789760793.985684:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.985705:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6936, 6936] 00000008:00010000:2.0:1789760793.985722:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.985727:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760793.985738:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28409856, 28413952) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.985750:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28409856=0x1b18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.985808:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.985811:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28413952 00000020:00010000:2.0:1789760793.985894:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.985899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.985902:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.985906:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.985909:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.985915:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.985922:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.985946:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b240 00000008:00100000:2.0:1789760793.986178:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81880 - object 0x240000400:5 offset 28409856<>28413952 00000100:00100000:3.0:1789760793.986238:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874304:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.015193:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974255 gen 3 00000008:00010000:3.0:1789760794.015247:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b240 00000020:00200000:3.0:1789760794.015256:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.015269:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.015332:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b240 00000100:00100000:3.0:1789760794.015365:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874304:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.015611:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6936, 6936] 00000080:00200000:2.0:1789760794.015627:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.015632:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.015649:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28413952 i_blocks 57353 00000020:00200000:2.0:1789760794.015653:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.015665:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.015699:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28413952, count: 4096 00000080:00200000:2.0:1789760794.015706:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28413952, bytes: 4096 00000080:00200000:2.0:1789760794.015717:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.015726:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.015728:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28413952, 28418048) 0 0 00000080:00200000:2.0:1789760794.015733:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.015749:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6937, 6937] 00020000:00200000:2.0:1789760794.015757:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28413952 chunk: [28413952, 28418048) 28418048, 4096 00020000:00200000:2.0:1789760794.015763:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.015783:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6937, 6937] 00000008:00010000:2.0:1789760794.015801:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.015807:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.015877:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28413952, 28418048) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.015895:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28413952=0x1b19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.015964:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.015977:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28418048 00000020:00010000:2.0:1789760794.015986:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.015989:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.015992:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.016147:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.016151:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.016160:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.016168:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.016185:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b900 00000008:00100000:2.0:1789760794.016293:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f82a00 - object 0x240000400:5 offset 28413952<>28418048 00000100:00100000:3.0:1789760794.016362:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493874432:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.059785:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974256 gen 3 00000008:00010000:3.0:1789760794.059840:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b900 00000020:00200000:3.0:1789760794.059849:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.059862:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.059922:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b900 00000100:00100000:3.0:1789760794.059960:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493874432:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.060043:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6937, 6937] 00000080:00200000:2.0:1789760794.060061:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.060066:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.060082:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28418048 i_blocks 57353 00000020:00200000:2.0:1789760794.060086:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.060096:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.060132:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28418048, count: 4096 00000080:00200000:2.0:1789760794.060141:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28418048, bytes: 4096 00000080:00200000:2.0:1789760794.060150:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.060158:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.060161:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28418048, 28422144) 0 0 00000080:00200000:2.0:1789760794.060166:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.060182:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6938, 6938] 00020000:00200000:2.0:1789760794.060190:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28418048 chunk: [28418048, 28422144) 28422144, 4096 00020000:00200000:2.0:1789760794.060195:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.060215:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6938, 6938] 00000008:00010000:2.0:1789760794.060233:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.060239:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.060250:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28418048, 28422144) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.060263:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28418048=0x1b1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.060321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.060323:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28422144 00000020:00010000:2.0:1789760794.060333:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.060336:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.060339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.060342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.060345:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.060353:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.060360:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.060375:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b480 00000008:00100000:2.0:1789760794.060664:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 28418048<>28422144 00000100:00100000:3.0:1789760794.060720:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760794.060880:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493874560:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.096041:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974257 gen 3 00000008:00010000:3.0:1789760794.096095:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b480 00000020:00200000:3.0:1789760794.096104:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.096115:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.096138:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b480 00000100:00100000:3.0:1789760794.096166:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493874560:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760794.096279:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6938, 6938] 00000080:00200000:0.0:1789760794.096297:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.096302:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.096318:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28422144 i_blocks 57353 00000020:00200000:0.0:1789760794.096321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.096332:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.096367:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28422144, count: 4096 00000080:00200000:0.0:1789760794.096375:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28422144, bytes: 4096 00000080:00200000:0.0:1789760794.096387:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.096430:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.096432:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28422144, 28426240) 0 0 00000080:00200000:0.0:1789760794.096437:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.096451:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6939, 6939] 00020000:00200000:0.0:1789760794.096459:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28422144 chunk: [28422144, 28426240) 28426240, 4096 00020000:00200000:0.0:1789760794.096462:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.096478:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6939, 6939] 00000008:00010000:0.0:1789760794.096494:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.096500:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760794.096512:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28422144, 28426240) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.096525:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28422144=0x1b1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.096577:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.096579:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28426240 00000020:00010000:0.0:1789760794.096586:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.096588:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.096590:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.096593:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.096595:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.096601:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.096605:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.096617:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760794.096698:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e680 - object 0x240000400:5 offset 28422144<>28426240 00000100:00100000:1.0:1789760794.098145:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493874688:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760794.142845:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974258 gen 3 00000008:00010000:1.0:1789760794.142896:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760794.142904:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.142914:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.143004:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760794.143034:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493874688:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760794.143168:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6939, 6939] 00000080:00200000:0.0:1789760794.143184:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.143191:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.143207:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28426240 i_blocks 57353 00000020:00200000:0.0:1789760794.143211:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.143222:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.143260:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28426240, count: 4096 00000080:00200000:0.0:1789760794.143267:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28426240, bytes: 4096 00000080:00200000:0.0:1789760794.143277:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.143285:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.143287:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28426240, 28430336) 0 0 00000080:00200000:0.0:1789760794.143292:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.143308:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6940, 6940] 00020000:00200000:0.0:1789760794.143317:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28426240 chunk: [28426240, 28430336) 28430336, 4096 00020000:00200000:0.0:1789760794.143321:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.143341:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6940, 6940] 00000008:00010000:0.0:1789760794.143358:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.143364:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760794.143376:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28426240, 28430336) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.143389:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28426240=0x1b1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.143540:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.143544:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28430336 00000020:00010000:0.0:1789760794.143553:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.143557:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.143560:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.143563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.143566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.143574:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.143581:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.143597:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760794.143698:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f480 - object 0x240000400:5 offset 28426240<>28430336 00000100:00100000:1.0:1789760794.143765:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493874816:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760794.181706:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974259 gen 3 00000008:00010000:1.0:1789760794.181755:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760794.181765:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.181776:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.181804:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760794.181834:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493874816:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760794.181972:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6940, 6940] 00000080:00200000:3.0:1789760794.181989:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.181994:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.182019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28430336 i_blocks 57353 00000020:00200000:3.0:1789760794.182023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.182034:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.182068:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28430336, count: 4096 00000080:00200000:3.0:1789760794.182076:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28430336, bytes: 4096 00000080:00200000:3.0:1789760794.182090:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.182099:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760794.182101:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28430336, 28434432) 0 0 00000080:00200000:3.0:1789760794.182105:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.182122:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6941, 6941] 00020000:00200000:3.0:1789760794.182130:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28430336 chunk: [28430336, 28434432) 28434432, 4096 00020000:00200000:3.0:1789760794.182134:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.182158:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6941, 6941] 00000008:00010000:3.0:1789760794.182179:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.182185:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760794.182197:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28430336, 28434432) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.182211:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28430336=0x1b1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.182275:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.182278:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28434432 00000020:00010000:3.0:1789760794.182287:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.182291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.182294:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.182298:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.182301:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.182310:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.182317:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.182333:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760794.182488:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28430336<>28434432 00000100:00100000:2.0:1789760794.182947:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760794.182975:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874944:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.227389:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974260 gen 3 00000008:00010000:3.0:1789760794.227523:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:3.0:1789760794.227531:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.227542:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.227600:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:3.0:1789760794.227628:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493874944:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.227706:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6941, 6941] 00000080:00200000:2.0:1789760794.227723:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.227729:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.227744:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28434432 i_blocks 57353 00000020:00200000:2.0:1789760794.227748:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.227758:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.227791:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28434432, count: 4096 00000080:00200000:2.0:1789760794.227798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28434432, bytes: 4096 00000080:00200000:2.0:1789760794.227808:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.227876:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.227879:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28434432, 28438528) 0 0 00000080:00200000:2.0:1789760794.227885:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.227901:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6942, 6942] 00020000:00200000:2.0:1789760794.227910:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28434432 chunk: [28434432, 28438528) 28438528, 4096 00020000:00200000:2.0:1789760794.227914:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.227949:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6942, 6942] 00000008:00010000:2.0:1789760794.228084:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.228091:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.228105:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28434432, 28438528) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.228121:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28434432=0x1b1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.228182:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.228185:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28438528 00000020:00010000:2.0:1789760794.228195:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.228198:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.228200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.228204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.228206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.228214:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.228221:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.228236:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000008:00100000:2.0:1789760794.228338:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 28434432<>28438528 00000100:00100000:3.0:1789760794.228495:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875072:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760794.278582:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000020:00200000:3.0:1789760794.278734:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.278749:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.278895:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b3c0 00000100:00100000:3.0:1789760794.278934:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875072:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.282751:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6942, 6942] 00000080:00200000:2.0:1789760794.282840:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.282846:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.282864:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28438528 i_blocks 57353 00000020:00200000:2.0:1789760794.282870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.282888:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.283048:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28438528, count: 4096 00000080:00200000:2.0:1789760794.283057:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28438528, bytes: 4096 00000080:00200000:2.0:1789760794.283071:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.283080:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.283083:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28438528, 28442624) 0 0 00000080:00200000:2.0:1789760794.283087:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.283509:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6943, 6943] 00020000:00200000:2.0:1789760794.283521:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28438528 chunk: [28438528, 28442624) 28442624, 4096 00020000:00200000:2.0:1789760794.283525:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.283547:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6943, 6943] 00000008:00010000:2.0:1789760794.283566:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.283571:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.283583:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28438528, 28442624) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.283597:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28438528=0x1b1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.283659:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.283662:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28442624 00000020:00010000:2.0:1789760794.283670:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.283673:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.283677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.283680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.283683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.283691:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.283698:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.283720:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b540 00000008:00100000:2.0:1789760794.284257:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28438528<>28442624 00000100:00100000:3.0:1789760794.284362:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875200:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.323372:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974261 gen 3 00000100:00100000:3.0:1789760794.323464:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81180 x1876699493875200/t4294974262(4294974262) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760810 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760794.323515:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b540 00000020:00200000:3.0:1789760794.323523:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.323535:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.323557:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b540 00000100:00100000:3.0:1789760794.323583:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875200:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.324062:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6943, 6943] 00000080:00200000:2.0:1789760794.324079:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.324085:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.324101:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28442624 i_blocks 57353 00000020:00200000:2.0:1789760794.324105:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.324115:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.324149:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28442624, count: 4096 00000080:00200000:2.0:1789760794.324156:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28442624, bytes: 4096 00000080:00200000:2.0:1789760794.324167:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.324175:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.324177:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28442624, 28446720) 0 0 00000080:00200000:2.0:1789760794.324182:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.324198:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6944, 6944] 00020000:00200000:2.0:1789760794.324207:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28442624 chunk: [28442624, 28446720) 28446720, 4096 00020000:00200000:2.0:1789760794.324212:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.324232:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6944, 6944] 00000008:00010000:2.0:1789760794.324248:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.324254:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.324266:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28442624, 28446720) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.324280:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28442624=0x1b20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.324339:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.324342:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28446720 00000020:00010000:2.0:1789760794.324350:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.324354:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.324357:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.324361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.324364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.324371:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.324378:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.324437:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b300 00000008:00100000:2.0:1789760794.324537:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 28442624<>28446720 00000100:00100000:3.0:1789760794.324775:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875328:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.360247:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974263 gen 3 00000008:00010000:3.0:1789760794.360314:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b300 00000020:00200000:3.0:1789760794.360323:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.360334:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.360357:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b300 00000100:00100000:3.0:1789760794.360388:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875328:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.360613:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6944, 6944] 00000080:00200000:2.0:1789760794.360631:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.360638:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.360658:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28446720 i_blocks 57353 00000020:00200000:2.0:1789760794.360662:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.360673:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.360704:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28446720, count: 4096 00000080:00200000:2.0:1789760794.360724:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28446720, bytes: 4096 00000080:00200000:2.0:1789760794.360734:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.360743:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.360745:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28446720, 28450816) 0 0 00000080:00200000:2.0:1789760794.360749:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.360765:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6945, 6945] 00020000:00200000:2.0:1789760794.360776:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28446720 chunk: [28446720, 28450816) 28450816, 4096 00020000:00200000:2.0:1789760794.360782:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.360806:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6945, 6945] 00000008:00010000:2.0:1789760794.360830:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.360837:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.360851:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28446720, 28450816) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.360866:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28446720=0x1b21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.360933:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.360944:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28450816 00000020:00010000:2.0:1789760794.360954:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.360958:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.360961:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.360965:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.360967:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.360976:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.360984:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.361005:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b000 00000008:00100000:2.0:1789760794.361121:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 28446720<>28450816 00000100:00100000:3.0:1789760794.361460:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875456:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.396966:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974264 gen 3 00000008:00010000:3.0:1789760794.397017:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b000 00000020:00200000:3.0:1789760794.397025:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.397038:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.397090:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b000 00000100:00100000:3.0:1789760794.397119:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875456:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.397195:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6945, 6945] 00000080:00200000:2.0:1789760794.397211:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.397216:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.397233:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28450816 i_blocks 57353 00000020:00200000:2.0:1789760794.397237:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.397247:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.397283:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28450816, count: 4096 00000080:00200000:2.0:1789760794.397289:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28450816, bytes: 4096 00000080:00200000:2.0:1789760794.397298:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.397307:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.397309:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28450816, 28454912) 0 0 00000080:00200000:2.0:1789760794.397314:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.397331:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6946, 6946] 00020000:00200000:2.0:1789760794.397338:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28450816 chunk: [28450816, 28454912) 28454912, 4096 00020000:00200000:2.0:1789760794.397342:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.397362:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6946, 6946] 00000008:00010000:2.0:1789760794.397379:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.397386:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.397476:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28450816, 28454912) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.397491:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28450816=0x1b22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.397545:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.397548:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28454912 00000020:00010000:2.0:1789760794.397556:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.397559:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.397563:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.397566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.397569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.397577:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.397584:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.397598:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760794.397703:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 28450816<>28454912 00000100:00100000:3.0:1789760794.397778:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875584:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760794.426028:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760794.426045:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.426057:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.426077:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760794.426107:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875584:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.426199:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6946, 6946] 00000080:00200000:2.0:1789760794.426215:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.426221:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.426237:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28454912 i_blocks 57353 00000020:00200000:2.0:1789760794.426241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.426251:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.426284:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28454912, count: 4096 00000080:00200000:2.0:1789760794.426291:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28454912, bytes: 4096 00000080:00200000:2.0:1789760794.426300:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.426309:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.426311:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28454912, 28459008) 0 0 00000080:00200000:2.0:1789760794.426315:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.426332:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6947, 6947] 00020000:00200000:2.0:1789760794.426340:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28454912 chunk: [28454912, 28459008) 28459008, 4096 00020000:00200000:2.0:1789760794.426345:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.426365:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6947, 6947] 00000008:00010000:2.0:1789760794.426384:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.426390:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.426592:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28454912, 28459008) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.426608:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28454912=0x1b23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.426665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.426668:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28459008 00000020:00010000:2.0:1789760794.426677:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.426680:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.426683:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.426687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.426690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.426697:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.426704:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.426720:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760794.426823:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 28454912<>28459008 00000100:00100000:3.0:1789760794.426886:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875712:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.498066:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974266 gen 3 00000008:00010000:3.0:1789760794.498134:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760794.498143:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.498156:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.498178:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760794.498210:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875712:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.498356:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6947, 6947] 00000080:00200000:2.0:1789760794.498374:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.498380:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.498446:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28459008 i_blocks 57353 00000020:00200000:2.0:1789760794.498450:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.498463:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.498498:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28459008, count: 4096 00000080:00200000:2.0:1789760794.498505:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28459008, bytes: 4096 00000080:00200000:2.0:1789760794.498516:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.498526:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.498528:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28459008, 28463104) 0 0 00000080:00200000:2.0:1789760794.498533:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.498550:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6948, 6948] 00020000:00200000:2.0:1789760794.498558:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28459008 chunk: [28459008, 28463104) 28463104, 4096 00020000:00200000:2.0:1789760794.498563:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.498584:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6948, 6948] 00000008:00010000:2.0:1789760794.498603:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.498611:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.498623:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28459008, 28463104) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.498637:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28459008=0x1b24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.498696:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.498699:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28463104 00000020:00010000:2.0:1789760794.498707:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.498711:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.498714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.498717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.498720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.498728:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.498735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.498751:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760794.498860:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 28459008<>28463104 00000100:00100000:3.0:1789760794.499172:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875840:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760794.543500:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760794.543518:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.543531:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.543591:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760794.543622:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493875840:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.545539:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6948, 6948] 00000080:00200000:2.0:1789760794.545565:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.545572:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.545589:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28463104 i_blocks 57353 00000020:00200000:2.0:1789760794.545595:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.545610:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.545644:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28463104, count: 4096 00000080:00200000:2.0:1789760794.545653:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28463104, bytes: 4096 00000080:00200000:2.0:1789760794.545666:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.545675:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.545678:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28463104, 28467200) 0 0 00000080:00200000:2.0:1789760794.545683:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.545699:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6949, 6949] 00020000:00200000:2.0:1789760794.545707:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28463104 chunk: [28463104, 28467200) 28467200, 4096 00020000:00200000:2.0:1789760794.545713:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.545736:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6949, 6949] 00000008:00010000:2.0:1789760794.545755:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.545762:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.545775:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28463104, 28467200) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.545790:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28463104=0x1b25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.545859:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.545862:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28467200 00000020:00010000:2.0:1789760794.545871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.545874:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.545877:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.545881:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.545884:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.545892:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.545899:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.547197:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760794.547316:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 28463104<>28467200 00000100:00100000:3.0:1789760794.547426:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875968:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.593990:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974268 gen 3 00000008:00010000:3.0:1789760794.594050:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760794.594057:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.594066:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.594127:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760794.594161:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493875968:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.594366:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6949, 6949] 00000080:00200000:2.0:1789760794.594379:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.594384:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.594563:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28467200 i_blocks 57353 00000020:00200000:2.0:1789760794.594568:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.594577:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.594602:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28467200, count: 4096 00000080:00200000:2.0:1789760794.594608:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28467200, bytes: 4096 00000080:00200000:2.0:1789760794.594616:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.594623:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.594625:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28467200, 28471296) 0 0 00000080:00200000:2.0:1789760794.594628:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.594640:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6950, 6950] 00020000:00200000:2.0:1789760794.594646:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28467200 chunk: [28467200, 28471296) 28471296, 4096 00020000:00200000:2.0:1789760794.594649:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.594664:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6950, 6950] 00000008:00010000:2.0:1789760794.594675:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.594680:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.594688:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28467200, 28471296) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.594697:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28467200=0x1b26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.594738:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.594740:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28471296 00000020:00010000:2.0:1789760794.594747:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.594750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.594752:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.594756:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.594758:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.594857:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.594921:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.594947:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760794.595034:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 28467200<>28471296 00000100:00100000:3.0:1789760794.595098:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876096:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.672850:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974269 gen 3 00000008:00010000:3.0:1789760794.672963:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760794.672971:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.672980:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.673067:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760794.673090:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876096:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.673173:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6950, 6950] 00000080:00200000:2.0:1789760794.673186:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.673191:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.673202:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28471296 i_blocks 57353 00000020:00200000:2.0:1789760794.673206:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.673215:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.673253:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28471296, count: 4096 00000080:00200000:2.0:1789760794.673259:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28471296, bytes: 4096 00000080:00200000:2.0:1789760794.673267:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.673273:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.673275:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28471296, 28475392) 0 0 00000080:00200000:2.0:1789760794.673278:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.673291:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6951, 6951] 00020000:00200000:2.0:1789760794.673298:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28471296 chunk: [28471296, 28475392) 28475392, 4096 00020000:00200000:2.0:1789760794.673302:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.673318:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6951, 6951] 00000008:00010000:2.0:1789760794.673332:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.673336:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.673345:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28471296, 28475392) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.673355:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28471296=0x1b27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.673425:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.673428:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28475392 00000020:00010000:2.0:1789760794.673434:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.673438:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.673441:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.673444:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.673446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.673453:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.673458:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.673473:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760794.673673:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81500 - object 0x240000400:5 offset 28471296<>28475392 00000100:00100000:3.0:1789760794.673776:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876224:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.779083:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974270 gen 3 00000008:00010000:3.0:1789760794.779146:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760794.779155:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.779167:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.779186:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760794.779217:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876224:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.779275:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6951, 6951] 00000080:00200000:2.0:1789760794.779298:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.779305:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.779317:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28475392 i_blocks 57353 00000020:00200000:2.0:1789760794.779322:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.779335:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.779364:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28475392, count: 4096 00000080:00200000:2.0:1789760794.779371:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28475392, bytes: 4096 00000080:00200000:2.0:1789760794.779378:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.779386:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.779388:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28475392, 28479488) 0 0 00000080:00200000:2.0:1789760794.779497:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.779513:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6952, 6952] 00020000:00200000:2.0:1789760794.779520:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28475392 chunk: [28475392, 28479488) 28479488, 4096 00020000:00200000:2.0:1789760794.779523:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.779627:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6952, 6952] 00000008:00010000:2.0:1789760794.779643:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.779648:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.779657:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28475392, 28479488) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.779669:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28475392=0x1b28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.779719:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.779722:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28479488 00000020:00010000:2.0:1789760794.779729:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.779732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.779734:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.779738:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.779741:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.779747:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.779753:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.779772:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760794.779853:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80380 - object 0x240000400:5 offset 28475392<>28479488 00000100:00100000:3.0:1789760794.780094:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876352:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760794.861509:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760794.912518:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760794.912535:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.912548:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.912604:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760794.912632:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876352:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.912910:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6952, 6952] 00000080:00200000:2.0:1789760794.912925:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.912938:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.912953:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28479488 i_blocks 57353 00000020:00200000:2.0:1789760794.912957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.912967:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.913176:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28479488, count: 4096 00000080:00200000:2.0:1789760794.913184:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28479488, bytes: 4096 00000080:00200000:2.0:1789760794.913193:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.913202:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.913204:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28479488, 28483584) 0 0 00000080:00200000:2.0:1789760794.913209:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.913224:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6953, 6953] 00020000:00200000:2.0:1789760794.913231:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28479488 chunk: [28479488, 28483584) 28483584, 4096 00020000:00200000:2.0:1789760794.913236:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.913253:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6953, 6953] 00000008:00010000:2.0:1789760794.913270:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.913275:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.913286:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28479488, 28483584) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.913298:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28479488=0x1b29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.913352:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.913354:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28483584 00000020:00010000:2.0:1789760794.913361:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.913365:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.913368:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.913371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.913374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.913381:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.913387:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.913481:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bd80 00000008:00100000:2.0:1789760794.913580:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80000 - object 0x240000400:5 offset 28479488<>28483584 00000100:00100000:3.0:1789760794.913641:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876480:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760794.943473:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974272 gen 3 00000008:00010000:3.0:1789760794.943549:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bd80 00000020:00200000:3.0:1789760794.943558:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.943568:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.943586:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bd80 00000100:00100000:3.0:1789760794.943613:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876480:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.943741:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6953, 6953] 00000080:00200000:2.0:1789760794.943760:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.943767:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.943784:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28483584 i_blocks 57353 00000020:00200000:2.0:1789760794.943788:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.943799:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.943842:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28483584, count: 4096 00000080:00200000:2.0:1789760794.943853:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28483584, bytes: 4096 00000080:00200000:2.0:1789760794.943864:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.943874:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.943877:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28483584, 28487680) 0 0 00000080:00200000:2.0:1789760794.943881:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.943899:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6954, 6954] 00020000:00200000:2.0:1789760794.943906:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28483584 chunk: [28483584, 28487680) 28487680, 4096 00020000:00200000:2.0:1789760794.943911:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.943930:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6954, 6954] 00000008:00010000:2.0:1789760794.943956:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.943964:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.943976:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28483584, 28487680) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.943999:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28483584=0x1b2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.944067:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.944070:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28487680 00000020:00010000:2.0:1789760794.944079:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.944082:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.944085:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.944089:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.944092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.944099:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.944106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.944122:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bb40 00000008:00100000:2.0:1789760794.944229:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80a80 - object 0x240000400:5 offset 28483584<>28487680 00000100:00100000:3.0:1789760794.944535:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876608:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760794.995676:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bb40 00000020:00200000:3.0:1789760794.995696:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.995714:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.995738:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bb40 00000100:00100000:3.0:1789760794.995771:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493876608:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760794.995957:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6954, 6954] 00000080:00200000:2.0:1789760794.995976:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.995983:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.995999:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28487680 i_blocks 57353 00000020:00200000:2.0:1789760794.996003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.996015:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.996064:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28487680, count: 4096 00000080:00200000:2.0:1789760794.996072:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28487680, bytes: 4096 00000080:00200000:2.0:1789760794.996082:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.996091:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.996093:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28487680, 28491776) 0 0 00000080:00200000:2.0:1789760794.996098:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.996114:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6955, 6955] 00020000:00200000:2.0:1789760794.996124:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28487680 chunk: [28487680, 28491776) 28491776, 4096 00020000:00200000:2.0:1789760794.996128:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.996151:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6955, 6955] 00000008:00010000:2.0:1789760794.996169:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.996176:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760794.996188:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28487680, 28491776) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.996204:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28487680=0x1b2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.996267:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.996269:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28491776 00000020:00010000:2.0:1789760794.996278:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.996281:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.996284:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.996288:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.996291:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.996298:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.996305:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.996322:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000008:00100000:2.0:1789760794.996474:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81f80 - object 0x240000400:5 offset 28487680<>28491776 00000100:00100000:3.0:1789760794.997261:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876736:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760795.040704:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974273 gen 3 00000100:00100000:3.0:1789760795.040744:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81f80 x1876699493876736/t4294974274(4294974274) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760810 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760795.041368:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000020:00200000:3.0:1789760795.041376:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.041388:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.041458:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3bcc0 00000100:00100000:3.0:1789760795.041487:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493876736:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.041651:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6955, 6955] 00000080:00200000:0.0:1789760795.041667:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.041674:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.041690:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28491776 i_blocks 57353 00000020:00200000:0.0:1789760795.041694:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.041705:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.041739:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28491776, count: 4096 00000080:00200000:0.0:1789760795.041747:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28491776, bytes: 4096 00000080:00200000:0.0:1789760795.041767:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.041776:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.041779:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28491776, 28495872) 0 0 00000080:00200000:0.0:1789760795.041783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.041800:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6956, 6956] 00020000:00200000:0.0:1789760795.041809:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28491776 chunk: [28491776, 28495872) 28495872, 4096 00020000:00200000:0.0:1789760795.041814:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.041833:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6956, 6956] 00000008:00010000:0.0:1789760795.042018:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.042025:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.042038:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28491776, 28495872) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.042055:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28491776=0x1b2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.042186:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.042189:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28495872 00000020:00010000:0.0:1789760795.042199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.042203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.042206:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.042209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.042212:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.042220:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.042228:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.042246:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760795.042457:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d880 - object 0x240000400:5 offset 28491776<>28495872 00000100:00100000:1.0:1789760795.042591:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493876864:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.105665:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974275 gen 3 00000008:00010000:1.0:1789760795.105732:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760795.105741:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.105752:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.105773:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760795.105802:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493876864:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.105898:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6956, 6956] 00000080:00200000:0.0:1789760795.105915:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.105920:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.105946:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28495872 i_blocks 57353 00000020:00200000:0.0:1789760795.105950:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.105960:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.105993:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28495872, count: 4096 00000080:00200000:0.0:1789760795.106001:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28495872, bytes: 4096 00000080:00200000:0.0:1789760795.106010:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.106018:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.106020:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28495872, 28499968) 0 0 00000080:00200000:0.0:1789760795.106024:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.106041:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6957, 6957] 00020000:00200000:0.0:1789760795.106050:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28495872 chunk: [28495872, 28499968) 28499968, 4096 00020000:00200000:0.0:1789760795.106055:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.106073:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6957, 6957] 00000008:00010000:0.0:1789760795.106090:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.106096:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.106119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28495872, 28499968) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.106133:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28495872=0x1b2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.106189:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.106192:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28499968 00000020:00010000:0.0:1789760795.106200:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.106204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.106207:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.106211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.106214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.106222:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.106229:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.106245:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760795.106357:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ce00 - object 0x240000400:5 offset 28495872<>28499968 00000100:00100000:1.0:1789760795.106597:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760795.106616:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493876992:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760795.182339:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760795.182354:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.182365:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.182439:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760795.182466:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493876992:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.182634:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6957, 6957] 00000080:00200000:0.0:1789760795.182652:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.182659:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.182675:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28499968 i_blocks 57353 00000020:00200000:0.0:1789760795.182679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.182691:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.182731:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28499968, count: 4096 00000080:00200000:0.0:1789760795.182738:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28499968, bytes: 4096 00000080:00200000:0.0:1789760795.182752:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.182762:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.182765:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28499968, 28504064) 0 0 00000080:00200000:0.0:1789760795.182770:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.182790:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6958, 6958] 00020000:00200000:0.0:1789760795.182801:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28499968 chunk: [28499968, 28504064) 28504064, 4096 00020000:00200000:0.0:1789760795.182805:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.182826:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6958, 6958] 00000008:00010000:0.0:1789760795.182846:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.182854:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.182866:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28499968, 28504064) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.182883:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28499968=0x1b2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.182956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.182959:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28504064 00000020:00010000:0.0:1789760795.182968:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.182971:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.182975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.182978:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.182981:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.182988:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.182995:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.183011:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760795.183124:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304f800 - object 0x240000400:5 offset 28499968<>28504064 00000100:00100000:1.0:1789760795.183293:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877120:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.223910:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974276 gen 3 00000100:00100000:1.0:1789760795.223964:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304f800 x1876699493877120/t4294974277(4294974277) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760795.224022:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760795.224029:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.224039:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.224066:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760795.224091:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877120:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.224285:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6958, 6958] 00000080:00200000:0.0:1789760795.224302:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.224308:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.224323:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28504064 i_blocks 57353 00000020:00200000:0.0:1789760795.224326:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.224338:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.224372:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28504064, count: 4096 00000080:00200000:0.0:1789760795.224380:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28504064, bytes: 4096 00000080:00200000:0.0:1789760795.224389:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.224435:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.224437:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28504064, 28508160) 0 0 00000080:00200000:0.0:1789760795.224442:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.224459:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6959, 6959] 00020000:00200000:0.0:1789760795.224468:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28504064 chunk: [28504064, 28508160) 28508160, 4096 00020000:00200000:0.0:1789760795.224473:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.224492:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6959, 6959] 00000008:00010000:0.0:1789760795.224513:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.224519:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.224531:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28504064, 28508160) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.224546:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28504064=0x1b2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.226210:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.226222:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28508160 00000020:00010000:0.0:1789760795.226236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.226241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.226245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.226249:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.226252:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.226261:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.226270:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.226296:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760795.226540:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d500 - object 0x240000400:5 offset 28504064<>28508160 00000100:00100000:1.0:1789760795.226612:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493877248:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.278805:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974278 gen 3 00000008:00010000:1.0:1789760795.278875:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760795.278884:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.278893:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.278960:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760795.278989:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493877248:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.279218:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6959, 6959] 00000080:00200000:3.0:1789760795.279236:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.279242:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.279258:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28508160 i_blocks 57353 00000020:00200000:3.0:1789760795.279262:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.279272:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.279309:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28508160, count: 4096 00000080:00200000:3.0:1789760795.279316:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28508160, bytes: 4096 00000080:00200000:3.0:1789760795.279327:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.279336:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.279338:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28508160, 28512256) 0 0 00000080:00200000:3.0:1789760795.279343:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.279363:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6960, 6960] 00020000:00200000:3.0:1789760795.279371:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28508160 chunk: [28508160, 28512256) 28512256, 4096 00020000:00200000:3.0:1789760795.279376:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.279500:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6960, 6960] 00000008:00010000:3.0:1789760795.279522:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.279528:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.279541:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28508160, 28512256) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.279556:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28508160=0x1b30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.281586:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.281599:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28512256 00000020:00010000:3.0:1789760795.281617:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.281621:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.281624:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.281629:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.281631:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.281645:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.281656:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.281784:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760795.281916:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28508160<>28512256 00000100:00100000:2.0:1789760795.282044:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493877376:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.337474:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974279 gen 3 00000008:00010000:2.0:1789760795.337538:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760795.337547:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.337558:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.337593:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760795.337625:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493877376:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.337756:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6960, 6960] 00000080:00200000:3.0:1789760795.337775:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.337781:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.337816:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28512256 i_blocks 57353 00000020:00200000:3.0:1789760795.337820:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.337830:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.337865:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28512256, count: 4096 00000080:00200000:3.0:1789760795.337872:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28512256, bytes: 4096 00000080:00200000:3.0:1789760795.337885:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.337894:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.337897:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28512256, 28516352) 0 0 00000080:00200000:3.0:1789760795.337901:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.337936:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6961, 6961] 00020000:00200000:3.0:1789760795.337945:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28512256 chunk: [28512256, 28516352) 28516352, 4096 00020000:00200000:3.0:1789760795.337950:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.337969:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6961, 6961] 00000008:00010000:3.0:1789760795.337987:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.337993:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.338005:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28512256, 28516352) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.338017:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28512256=0x1b31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.338069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.338072:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28516352 00000020:00010000:3.0:1789760795.338080:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.338083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.338086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.338090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.338092:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.338099:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.338106:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.338120:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760795.338214:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 28512256<>28516352 00000100:00100000:2.0:1789760795.338669:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493877504:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.386069:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974280 gen 3 00000008:00010000:2.0:1789760795.386182:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760795.386191:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.386203:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.386272:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760795.386310:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493877504:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.386557:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6961, 6961] 00000080:00200000:3.0:1789760795.386583:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.386589:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.386606:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28516352 i_blocks 57353 00000020:00200000:3.0:1789760795.386610:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.386620:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.386771:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28516352, count: 4096 00000080:00200000:3.0:1789760795.386782:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28516352, bytes: 4096 00000080:00200000:3.0:1789760795.386796:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.386806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.386808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28516352, 28520448) 0 0 00000080:00200000:3.0:1789760795.386813:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.386838:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6962, 6962] 00020000:00200000:3.0:1789760795.386846:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28516352 chunk: [28516352, 28520448) 28520448, 4096 00020000:00200000:3.0:1789760795.386851:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.386872:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6962, 6962] 00000008:00010000:3.0:1789760795.386891:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.386897:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.386908:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28516352, 28520448) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.386921:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28516352=0x1b32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.386987:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.386990:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28520448 00000020:00010000:3.0:1789760795.386998:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.387002:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.387005:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.387009:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.387012:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.387020:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.387028:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.387046:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760795.387230:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 28516352<>28520448 00000100:00100000:2.0:1789760795.387308:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493877632:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760795.420021:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760795.420038:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.420049:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.420363:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760795.420442:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493877632:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.423549:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6962, 6962] 00000080:00200000:0.0:1789760795.423578:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.423584:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.423666:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28520448 i_blocks 57353 00000020:00200000:0.0:1789760795.423674:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.423745:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.423779:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28520448, count: 4096 00000080:00200000:0.0:1789760795.423790:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28520448, bytes: 4096 00000080:00200000:0.0:1789760795.423807:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.423816:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.423819:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28520448, 28524544) 0 0 00000080:00200000:0.0:1789760795.423823:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.423839:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6963, 6963] 00020000:00200000:0.0:1789760795.423847:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28520448 chunk: [28520448, 28524544) 28524544, 4096 00020000:00200000:0.0:1789760795.423852:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.423874:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6963, 6963] 00000008:00010000:0.0:1789760795.423892:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.423899:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.423912:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28520448, 28524544) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.423937:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28520448=0x1b33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.424000:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.424003:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28524544 00000020:00010000:0.0:1789760795.424011:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.424015:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.424019:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.424023:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.424026:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.424035:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.424042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.424061:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760795.424191:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304dc00 - object 0x240000400:5 offset 28520448<>28524544 00000100:00100000:1.0:1789760795.425014:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877760:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.471882:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974282 gen 3 00000008:00010000:1.0:1789760795.471953:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760795.471962:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.471972:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.471997:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760795.472026:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877760:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.472713:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6963, 6963] 00000080:00200000:0.0:1789760795.472731:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.472737:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.472753:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28524544 i_blocks 57353 00000020:00200000:0.0:1789760795.472758:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.472769:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.472805:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28524544, count: 4096 00000080:00200000:0.0:1789760795.472812:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28524544, bytes: 4096 00000080:00200000:0.0:1789760795.472824:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.472832:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.472834:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28524544, 28528640) 0 0 00000080:00200000:0.0:1789760795.472838:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.472852:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6964, 6964] 00020000:00200000:0.0:1789760795.472861:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28524544 chunk: [28524544, 28528640) 28528640, 4096 00020000:00200000:0.0:1789760795.472866:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.472884:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6964, 6964] 00000008:00010000:0.0:1789760795.472900:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.472907:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.472918:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28524544, 28528640) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.472941:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28524544=0x1b34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.473003:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.473006:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28528640 00000020:00010000:0.0:1789760795.473014:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.473018:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.473021:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.473024:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.473028:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.473036:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.473042:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.473058:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760795.473362:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 28524544<>28528640 00000100:00100000:1.0:1789760795.473533:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760795.473556:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877888:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.514064:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974283 gen 3 00000008:00010000:1.0:1789760795.514134:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760795.514143:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.514155:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.514169:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760795.514199:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493877888:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.514421:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6964, 6964] 00000080:00200000:0.0:1789760795.514440:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.514446:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.514462:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28528640 i_blocks 57353 00000020:00200000:0.0:1789760795.514466:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.514477:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.514510:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28528640, count: 4096 00000080:00200000:0.0:1789760795.514518:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28528640, bytes: 4096 00000080:00200000:0.0:1789760795.514529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.514538:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.514540:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28528640, 28532736) 0 0 00000080:00200000:0.0:1789760795.514545:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.514562:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6965, 6965] 00020000:00200000:0.0:1789760795.514572:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28528640 chunk: [28528640, 28532736) 28532736, 4096 00020000:00200000:0.0:1789760795.514576:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.514595:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6965, 6965] 00000008:00010000:0.0:1789760795.514612:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.514618:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.514629:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28528640, 28532736) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.514644:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28528640=0x1b35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.514702:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.514705:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28532736 00000020:00010000:0.0:1789760795.514714:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.514717:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.514720:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.514723:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.514726:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.514734:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.514741:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.514757:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760795.514869:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 28528640<>28532736 00000100:00100000:1.0:1789760795.515059:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493878016:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760795.556939:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974284 gen 3 00000008:00010000:1.0:1789760795.556999:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760795.557008:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.557019:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.557046:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760795.557089:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493878016:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.557285:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6965, 6965] 00000080:00200000:0.0:1789760795.557303:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.557310:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.557328:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28532736 i_blocks 57353 00000020:00200000:0.0:1789760795.557332:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.557345:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.557382:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28532736, count: 4096 00000080:00200000:0.0:1789760795.557390:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28532736, bytes: 4096 00000080:00200000:0.0:1789760795.557441:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.557451:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.557454:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28532736, 28536832) 0 0 00000080:00200000:0.0:1789760795.557459:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.557478:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6966, 6966] 00020000:00200000:0.0:1789760795.557488:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28532736 chunk: [28532736, 28536832) 28536832, 4096 00020000:00200000:0.0:1789760795.557493:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.557514:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6966, 6966] 00000008:00010000:0.0:1789760795.557536:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.557542:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.557556:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28532736, 28536832) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.557572:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28532736=0x1b36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.557637:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.557640:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28536832 00000020:00010000:0.0:1789760795.557650:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.557653:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.557657:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.557661:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.557664:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.557672:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.557679:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.557694:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760795.557794:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28532736<>28536832 00000100:00100000:0.0:1789760795.557840:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760795.557860:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493878144:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760795.609664:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974285 gen 3 00000008:00010000:0.0:1789760795.610089:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760795.610098:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.610109:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.610244:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760795.610281:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493878144:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.610496:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6966, 6966] 00000080:00200000:3.0:1789760795.610513:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.610518:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.610533:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28536832 i_blocks 57353 00000020:00200000:3.0:1789760795.610537:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.610548:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.610579:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28536832, count: 4096 00000080:00200000:3.0:1789760795.610586:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28536832, bytes: 4096 00000080:00200000:3.0:1789760795.610596:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.610606:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.610608:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28536832, 28540928) 0 0 00000080:00200000:3.0:1789760795.610612:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.610630:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6967, 6967] 00020000:00200000:3.0:1789760795.610638:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28536832 chunk: [28536832, 28540928) 28540928, 4096 00020000:00200000:3.0:1789760795.610642:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.610662:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6967, 6967] 00000008:00010000:3.0:1789760795.610681:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.610754:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.610769:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28536832, 28540928) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.610783:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28536832=0x1b37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.610956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.610960:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28540928 00000020:00010000:3.0:1789760795.610969:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.610973:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.610975:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.611039:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.611042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.611052:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.611059:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.611077:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760795.611250:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 28536832<>28540928 00000100:00100000:2.0:1789760795.611341:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878272:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.669656:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974286 gen 3 00000008:00010000:2.0:1789760795.670120:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760795.670130:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.670435:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.670535:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760795.670572:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878272:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.674263:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6967, 6967] 00000080:00200000:3.0:1789760795.674290:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.674297:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.674316:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28540928 i_blocks 57353 00000020:00200000:3.0:1789760795.674321:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.674337:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.674370:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28540928, count: 4096 00000080:00200000:3.0:1789760795.674378:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28540928, bytes: 4096 00000080:00200000:3.0:1789760795.674762:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.674775:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.674777:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28540928, 28545024) 0 0 00000080:00200000:3.0:1789760795.674783:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.674802:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6968, 6968] 00020000:00200000:3.0:1789760795.674812:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28540928 chunk: [28540928, 28545024) 28545024, 4096 00020000:00200000:3.0:1789760795.674816:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.674839:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6968, 6968] 00000008:00010000:3.0:1789760795.674859:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.674865:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.674877:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28540928, 28545024) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.674892:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28540928=0x1b38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.675296:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.675300:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28545024 00000020:00010000:3.0:1789760795.675311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.675314:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.675317:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.675321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.675324:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.675334:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.675342:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.679520:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760795.679663:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 28540928<>28545024 00000100:00100000:2.0:1789760795.679857:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.679885:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878400:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.733720:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974287 gen 3 00000008:00010000:2.0:1789760795.733781:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760795.733790:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.733801:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.733828:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760795.733856:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878400:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.734106:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6968, 6968] 00000080:00200000:3.0:1789760795.734122:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.734128:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.734145:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28545024 i_blocks 57353 00000020:00200000:3.0:1789760795.734148:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.734158:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.734191:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28545024, count: 4096 00000080:00200000:3.0:1789760795.734198:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28545024, bytes: 4096 00000080:00200000:3.0:1789760795.734208:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.734217:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.734219:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28545024, 28549120) 0 0 00000080:00200000:3.0:1789760795.734224:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.734241:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6969, 6969] 00020000:00200000:3.0:1789760795.734250:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28545024 chunk: [28545024, 28549120) 28549120, 4096 00020000:00200000:3.0:1789760795.734254:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.734273:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6969, 6969] 00000008:00010000:3.0:1789760795.734292:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.734299:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.734312:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28545024, 28549120) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.734326:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28545024=0x1b39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.734379:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.734382:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28549120 00000020:00010000:3.0:1789760795.734391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.734442:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.734446:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.734449:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.734453:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.734460:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.734467:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.734482:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760795.734588:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 28545024<>28549120 00000100:00100000:2.0:1789760795.734753:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878528:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.795775:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974288 gen 3 00000008:00010000:2.0:1789760795.795839:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760795.795847:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.795859:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.795879:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760795.795908:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878528:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.796319:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6969, 6969] 00000080:00200000:3.0:1789760795.796335:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.796341:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.796358:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28549120 i_blocks 57353 00000020:00200000:3.0:1789760795.796361:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.796382:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.796447:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28549120, count: 4096 00000080:00200000:3.0:1789760795.796455:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28549120, bytes: 4096 00000080:00200000:3.0:1789760795.796466:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.796476:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.796478:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28549120, 28553216) 0 0 00000080:00200000:3.0:1789760795.796484:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.796502:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6970, 6970] 00020000:00200000:3.0:1789760795.796511:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28549120 chunk: [28549120, 28553216) 28553216, 4096 00020000:00200000:3.0:1789760795.796516:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.796537:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6970, 6970] 00000008:00010000:3.0:1789760795.796558:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.796564:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.796576:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28549120, 28553216) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.796592:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28549120=0x1b3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.796651:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.796654:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28553216 00000020:00010000:3.0:1789760795.796662:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.796666:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.796669:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.796672:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.796674:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.796683:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.796690:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.796707:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760795.797126:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 28549120<>28553216 00000100:00100000:2.0:1789760795.797254:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493878656:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.838976:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974289 gen 3 00000008:00010000:2.0:1789760795.839037:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760795.839046:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.839059:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.839080:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760795.839110:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493878656:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.839314:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6970, 6970] 00000080:00200000:3.0:1789760795.839331:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.839337:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.839352:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28553216 i_blocks 57353 00000020:00200000:3.0:1789760795.839356:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.839366:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.839567:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28553216, count: 4096 00000080:00200000:3.0:1789760795.839576:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28553216, bytes: 4096 00000080:00200000:3.0:1789760795.839585:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.839594:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.839596:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28553216, 28557312) 0 0 00000080:00200000:3.0:1789760795.839601:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.839619:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6971, 6971] 00020000:00200000:3.0:1789760795.839627:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28553216 chunk: [28553216, 28557312) 28557312, 4096 00020000:00200000:3.0:1789760795.839632:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.839651:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6971, 6971] 00000008:00010000:3.0:1789760795.839672:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.839678:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.839689:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28553216, 28557312) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.839703:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28553216=0x1b3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.839761:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.839764:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28557312 00000020:00010000:3.0:1789760795.839772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.839776:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.839779:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.839782:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.839785:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.839793:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.839800:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.839817:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760795.839937:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148e00 - object 0x240000400:5 offset 28553216<>28557312 00000100:00100000:2.0:1789760795.840099:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878784:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760795.880685:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974290 gen 3 00000008:00010000:2.0:1789760795.880737:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760795.880841:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.880853:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.880877:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760795.880908:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878784:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760795.880979:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6971, 6971] 00000080:00200000:3.0:1789760795.880997:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760795.881004:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760795.881019:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28557312 i_blocks 57353 00000020:00200000:3.0:1789760795.881023:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760795.881130:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760795.881166:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28557312, count: 4096 00000080:00200000:3.0:1789760795.881176:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28557312, bytes: 4096 00000080:00200000:3.0:1789760795.881190:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760795.881200:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760795.881202:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28557312, 28561408) 0 0 00000080:00200000:3.0:1789760795.881206:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760795.881223:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6972, 6972] 00020000:00200000:3.0:1789760795.881231:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28557312 chunk: [28557312, 28561408) 28561408, 4096 00020000:00200000:3.0:1789760795.881236:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760795.881257:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6972, 6972] 00000008:00010000:3.0:1789760795.881280:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760795.881287:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760795.881299:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28557312, 28561408) append 0 append lockpos 0 00000080:00200000:3.0:1789760795.881314:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28557312=0x1b3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760795.881375:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760795.881379:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28561408 00000020:00010000:3.0:1789760795.881388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760795.881391:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.883311:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760795.883318:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760795.883321:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760795.883333:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760795.883345:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760795.883383:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760795.883649:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148700 - object 0x240000400:5 offset 28557312<>28561408 00000100:00100000:3.0:1789760795.883700:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.883724:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878912:192.168.203.160@tcp:4:dd.0 00010000:00010000:0.0:1789760795.885071:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:2.0:1789760795.924736:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760795.924750:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760795.924758:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760795.924777:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760795.924800:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493878912:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760795.924892:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6972, 6972] 00000080:00200000:0.0:1789760795.924909:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.924916:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.924940:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28561408 i_blocks 57353 00000020:00200000:0.0:1789760795.924944:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.924955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.925002:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28561408, count: 4096 00000080:00200000:0.0:1789760795.925009:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28561408, bytes: 4096 00000080:00200000:0.0:1789760795.925018:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.925026:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.925029:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28561408, 28565504) 0 0 00000080:00200000:0.0:1789760795.925034:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.925051:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6973, 6973] 00020000:00200000:0.0:1789760795.925059:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28561408 chunk: [28561408, 28565504) 28565504, 4096 00020000:00200000:0.0:1789760795.925063:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.925082:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6973, 6973] 00000008:00010000:0.0:1789760795.925101:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.925107:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760795.925119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28561408, 28565504) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.925133:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28561408=0x1b3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.925185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.925187:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28565504 00000020:00010000:0.0:1789760795.925194:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.925197:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.925199:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.925201:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.925203:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.925208:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.925214:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.925227:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760795.925326:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28561408<>28565504 00000100:00100000:0.0:1789760795.925433:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879040:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760795.956379:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974291 gen 3 00000100:00100000:0.0:1789760795.956576:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64304d180 x1876699493879040/t4294974292(4294974292) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760795.956629:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760795.956636:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760795.956646:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760795.956671:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760795.956696:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879040:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760795.956789:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6973, 6973] 00000080:00200000:2.0:1789760795.956805:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.956812:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.956831:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28565504 i_blocks 57353 00000020:00200000:2.0:1789760795.956835:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.956846:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.956882:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28565504, count: 4096 00000080:00200000:2.0:1789760795.956889:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28565504, bytes: 4096 00000080:00200000:2.0:1789760795.956900:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.956908:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.956910:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28565504, 28569600) 0 0 00000080:00200000:2.0:1789760795.956915:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.956942:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6974, 6974] 00020000:00200000:2.0:1789760795.956951:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28565504 chunk: [28565504, 28569600) 28569600, 4096 00020000:00200000:2.0:1789760795.956956:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.956976:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6974, 6974] 00000008:00010000:2.0:1789760795.957054:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.957061:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760795.957074:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28565504, 28569600) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.957089:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28565504=0x1b3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.957293:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.957296:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28569600 00000020:00010000:2.0:1789760795.957305:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.957309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.957312:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.957315:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.957319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.957326:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.957333:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.957351:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3ba80 00000008:00100000:2.0:1789760795.957711:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83b80 - object 0x240000400:5 offset 28565504<>28569600 00000100:00100000:3.0:1789760795.957792:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493879168:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760796.012594:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974292 gen 3 00000100:00100000:3.0:1789760796.012638:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f83b80 x1876699493879168/t4294974293(4294974293) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760796.012741:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3ba80 00000020:00200000:3.0:1789760796.012749:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760796.012761:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760796.012789:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3ba80 00000100:00100000:3.0:1789760796.012809:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493879168:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.015037:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6974, 6974] 00000080:00200000:0.0:1789760796.015067:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.015075:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.015095:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28569600 i_blocks 57353 00000020:00200000:0.0:1789760796.015100:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.015116:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.015152:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28569600, count: 4096 00000080:00200000:0.0:1789760796.015161:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28569600, bytes: 4096 00000080:00200000:0.0:1789760796.015176:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.015185:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.015187:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28569600, 28573696) 0 0 00000080:00200000:0.0:1789760796.015193:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.015210:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6975, 6975] 00020000:00200000:0.0:1789760796.015218:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28569600 chunk: [28569600, 28573696) 28573696, 4096 00020000:00200000:0.0:1789760796.015223:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.015244:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6975, 6975] 00000008:00010000:0.0:1789760796.015262:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.015269:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.015281:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28569600, 28573696) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.015298:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28569600=0x1b3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.015362:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.015365:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28573696 00000020:00010000:0.0:1789760796.015374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.015378:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.015381:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.015385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.015388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.015455:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.015463:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.015483:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760796.015781:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 28569600<>28573696 00000100:00100000:0.0:1789760796.015832:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760796.015857:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879296:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760796.061230:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974294 gen 3 00000008:00010000:0.0:1789760796.061298:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:0.0:1789760796.061305:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.061313:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.061377:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:0.0:1789760796.061432:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879296:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760796.061723:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6975, 6975] 00000080:00200000:2.0:1789760796.061739:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.061847:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.061865:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28573696 i_blocks 57353 00000020:00200000:2.0:1789760796.061870:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.061883:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.061917:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28573696, count: 4096 00000080:00200000:2.0:1789760796.061963:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28573696, bytes: 4096 00000080:00200000:2.0:1789760796.061978:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.061988:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.061990:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28573696, 28577792) 0 0 00000080:00200000:2.0:1789760796.061995:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.062012:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6976, 6976] 00020000:00200000:2.0:1789760796.062020:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28573696 chunk: [28573696, 28577792) 28577792, 4096 00020000:00200000:2.0:1789760796.062024:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.062046:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6976, 6976] 00000008:00010000:2.0:1789760796.062065:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.062071:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760796.062083:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28573696, 28577792) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.062097:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28573696=0x1b40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.062156:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.062159:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28577792 00000020:00010000:2.0:1789760796.062168:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.062172:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.062175:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.062178:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.062181:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.062188:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.062195:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.062213:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760796.062315:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83800 - object 0x240000400:5 offset 28573696<>28577792 00000100:00100000:3.0:1789760796.062422:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493879424:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760796.108651:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760796.108667:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.108679:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.108704:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760796.108735:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493879424:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.109144:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6976, 6976] 00000080:00200000:0.0:1789760796.109162:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.109168:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.109182:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28577792 i_blocks 57353 00000020:00200000:0.0:1789760796.109185:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.109196:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.109229:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28577792, count: 4096 00000080:00200000:0.0:1789760796.109237:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28577792, bytes: 4096 00000080:00200000:0.0:1789760796.109249:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.109257:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.109259:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28577792, 28581888) 0 0 00000080:00200000:0.0:1789760796.109264:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.109279:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6977, 6977] 00020000:00200000:0.0:1789760796.109287:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28577792 chunk: [28577792, 28581888) 28581888, 4096 00020000:00200000:0.0:1789760796.109291:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.109309:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6977, 6977] 00000008:00010000:0.0:1789760796.109326:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.109333:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.109344:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28577792, 28581888) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.109357:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28577792=0x1b41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.111175:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.111188:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28581888 00000020:00010000:0.0:1789760796.111204:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.111208:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.111213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.111217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.111220:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.111231:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.111240:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.111256:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760796.111367:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 28577792<>28581888 00000100:00100000:1.0:1789760796.111540:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879552:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760796.171136:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974296 gen 3 00000008:00010000:1.0:1789760796.171207:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760796.171215:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.171225:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.171251:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760796.171283:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879552:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.171630:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6977, 6977] 00000080:00200000:0.0:1789760796.171649:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.171655:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.171671:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28581888 i_blocks 57353 00000020:00200000:0.0:1789760796.171675:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.171691:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.171728:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28581888, count: 4096 00000080:00200000:0.0:1789760796.171736:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28581888, bytes: 4096 00000080:00200000:0.0:1789760796.171747:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.171757:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.171759:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28581888, 28585984) 0 0 00000080:00200000:0.0:1789760796.171765:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.171782:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6978, 6978] 00020000:00200000:0.0:1789760796.171791:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28581888 chunk: [28581888, 28585984) 28585984, 4096 00020000:00200000:0.0:1789760796.171795:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.171815:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6978, 6978] 00000008:00010000:0.0:1789760796.171833:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.171839:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.171852:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28581888, 28585984) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.171868:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28581888=0x1b42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.171941:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.171943:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28585984 00000020:00010000:0.0:1789760796.171953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.171957:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.171960:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.171964:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.171967:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.171985:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.171993:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.172011:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760796.172119:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 28581888<>28585984 00000100:00100000:1.0:1789760796.172282:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760796.172304:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879680:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760796.211790:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974297 gen 3 00000008:00010000:1.0:1789760796.211852:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760796.211862:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.211872:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.211893:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760796.211933:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493879680:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.212017:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6978, 6978] 00000080:00200000:0.0:1789760796.212033:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.212039:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.212054:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28585984 i_blocks 57353 00000020:00200000:0.0:1789760796.212058:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.212068:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.212105:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28585984, count: 4096 00000080:00200000:0.0:1789760796.212113:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28585984, bytes: 4096 00000080:00200000:0.0:1789760796.212123:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.212132:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.212134:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28585984, 28590080) 0 0 00000080:00200000:0.0:1789760796.212139:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.212155:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6979, 6979] 00020000:00200000:0.0:1789760796.212163:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28585984 chunk: [28585984, 28590080) 28590080, 4096 00020000:00200000:0.0:1789760796.212168:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.212193:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6979, 6979] 00000008:00010000:0.0:1789760796.212212:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.212219:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.212230:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28585984, 28590080) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.212245:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28585984=0x1b43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.212298:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.212301:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28590080 00000020:00010000:0.0:1789760796.212309:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.212313:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.212316:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.212319:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.212322:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.212329:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.212336:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.212351:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760796.212494:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 28585984<>28590080 00000100:00100000:0.0:1789760796.212526:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760796.212542:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493879808:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760796.256483:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974298 gen 3 00000008:00010000:0.0:1789760796.256537:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:0.0:1789760796.256547:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.256559:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.256584:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000080:00200000:2.0:1789760796.256621:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6979, 6979] 00000100:00100000:0.0:1789760796.256622:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493879808:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760796.256637:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.256643:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.256659:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28590080 i_blocks 57353 00000020:00200000:2.0:1789760796.256663:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.256674:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.256715:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28590080, count: 4096 00000080:00200000:2.0:1789760796.256722:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28590080, bytes: 4096 00000080:00200000:2.0:1789760796.256731:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.256739:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.256742:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28590080, 28594176) 0 0 00000080:00200000:2.0:1789760796.256746:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.256761:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6980, 6980] 00020000:00200000:2.0:1789760796.256770:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28590080 chunk: [28590080, 28594176) 28594176, 4096 00020000:00200000:2.0:1789760796.256775:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.256794:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6980, 6980] 00000008:00010000:2.0:1789760796.256812:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.256829:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760796.256841:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28590080, 28594176) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.256856:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28590080=0x1b44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.256909:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.256911:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28594176 00000020:00010000:2.0:1789760796.256918:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.256931:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.256934:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.256938:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.256941:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.256951:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.256957:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.256971:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760796.257078:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28590080<>28594176 00000100:00100000:2.0:1789760796.257124:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.257140:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493879936:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.316167:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974299 gen 3 00000008:00010000:2.0:1789760796.316218:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760796.316226:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.316236:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.316330:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760796.316360:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493879936:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.320126:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6980, 6980] 00000080:00200000:0.0:1789760796.320153:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.320161:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.320182:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28594176 i_blocks 57353 00000020:00200000:0.0:1789760796.320188:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.320208:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.320241:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28594176, count: 4096 00000080:00200000:0.0:1789760796.320249:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28594176, bytes: 4096 00000080:00200000:0.0:1789760796.320264:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.320273:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.320276:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28594176, 28598272) 0 0 00000080:00200000:0.0:1789760796.320280:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.320296:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6981, 6981] 00020000:00200000:0.0:1789760796.320317:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28594176 chunk: [28594176, 28598272) 28598272, 4096 00020000:00200000:0.0:1789760796.320322:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.320343:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6981, 6981] 00000008:00010000:0.0:1789760796.320361:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.320368:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.320382:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28594176, 28598272) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.320526:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28594176=0x1b45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.320598:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.320601:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28598272 00000020:00010000:0.0:1789760796.320610:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.320613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.320616:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.320619:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.320622:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.320631:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.320638:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.320664:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760796.320911:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304e300 - object 0x240000400:5 offset 28594176<>28598272 00000100:00100000:1.0:1789760796.321020:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493880064:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760796.351476:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974300 gen 3 00000008:00010000:1.0:1789760796.351529:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760796.351537:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.351547:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.351570:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760796.351599:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493880064:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.353074:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6981, 6981] 00000080:00200000:3.0:1789760796.353099:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.353105:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.353122:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28598272 i_blocks 57353 00000020:00200000:3.0:1789760796.353127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.353140:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.353174:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28598272, count: 4096 00000080:00200000:3.0:1789760796.353182:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28598272, bytes: 4096 00000080:00200000:3.0:1789760796.353195:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.353203:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.353206:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28598272, 28602368) 0 0 00000080:00200000:3.0:1789760796.353210:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.353227:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6982, 6982] 00020000:00200000:3.0:1789760796.353235:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28598272 chunk: [28598272, 28602368) 28602368, 4096 00020000:00200000:3.0:1789760796.353239:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.353260:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6982, 6982] 00000008:00010000:3.0:1789760796.353277:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.353283:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.353294:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28598272, 28602368) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.353529:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28598272=0x1b46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.353589:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.353591:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28602368 00000020:00010000:3.0:1789760796.353599:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.353603:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.353606:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.353609:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.353613:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.353620:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.353626:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.353643:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760796.354616:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514bb80 - object 0x240000400:5 offset 28598272<>28602368 00000100:00100000:2.0:1789760796.354693:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880192:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.400485:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974301 gen 3 00000008:00010000:2.0:1789760796.400546:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760796.400555:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.400566:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.400584:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760796.400619:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880192:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760796.400903:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6982, 6982] 00000080:00200000:0.0:1789760796.400929:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.400935:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.400953:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28602368 i_blocks 57353 00000020:00200000:0.0:1789760796.400956:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.400967:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.401005:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28602368, count: 4096 00000080:00200000:0.0:1789760796.401013:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28602368, bytes: 4096 00000080:00200000:0.0:1789760796.401023:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.401032:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.401035:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28602368, 28606464) 0 0 00000080:00200000:0.0:1789760796.401040:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.401058:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6983, 6983] 00020000:00200000:0.0:1789760796.401068:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28602368 chunk: [28602368, 28606464) 28606464, 4096 00020000:00200000:0.0:1789760796.401073:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.401098:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6983, 6983] 00000008:00010000:0.0:1789760796.401120:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.401127:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760796.401139:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28602368, 28606464) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.401153:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28602368=0x1b47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.401218:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.401221:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28606464 00000020:00010000:0.0:1789760796.401230:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.401233:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.401236:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.401239:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.401242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.401249:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.401257:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.401273:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb540 00000008:00100000:0.0:1789760796.401584:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304ed80 - object 0x240000400:5 offset 28602368<>28606464 00000100:00100000:1.0:1789760796.401637:0:2368:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760796.402794:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493880320:192.168.203.160@tcp:4:dd.0 00000008:00010000:1.0:1789760796.447222:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb540 00000020:00200000:1.0:1789760796.447237:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.447249:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.447279:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb540 00000100:00100000:1.0:1789760796.447314:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493880320:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760796.447472:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6983, 6983] 00000080:00200000:2.0:1789760796.447488:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.447494:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.447510:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28606464 i_blocks 57353 00000020:00200000:2.0:1789760796.447515:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.447525:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.447559:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28606464, count: 4096 00000080:00200000:2.0:1789760796.447566:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28606464, bytes: 4096 00000080:00200000:2.0:1789760796.447577:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.447585:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.447588:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28606464, 28610560) 0 0 00000080:00200000:2.0:1789760796.447592:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.447608:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6984, 6984] 00020000:00200000:2.0:1789760796.447617:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28606464 chunk: [28606464, 28610560) 28610560, 4096 00020000:00200000:2.0:1789760796.447621:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.447640:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6984, 6984] 00000008:00010000:2.0:1789760796.447750:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.447757:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760796.447769:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28606464, 28610560) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.447783:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28606464=0x1b48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.447854:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.447857:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28610560 00000020:00010000:2.0:1789760796.447865:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.447868:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.447871:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.447875:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.447878:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.447885:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.447892:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.447908:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760796.448330:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28606464<>28610560 00000100:00100000:2.0:1789760796.448379:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.448720:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880448:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.479896:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974302 gen 3 00000100:00100000:2.0:1789760796.479945:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81180 x1876699493880448/t4294974303(4294974303) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.480003:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760796.480011:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.480021:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.480041:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760796.480063:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880448:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.480197:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6984, 6984] 00000080:00200000:3.0:1789760796.480212:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.480218:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.480234:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28610560 i_blocks 57353 00000020:00200000:3.0:1789760796.480238:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.480248:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.480283:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28610560, count: 4096 00000080:00200000:3.0:1789760796.480301:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28610560, bytes: 4096 00000080:00200000:3.0:1789760796.480313:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.480322:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.480324:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28610560, 28614656) 0 0 00000080:00200000:3.0:1789760796.480328:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.480345:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6985, 6985] 00020000:00200000:3.0:1789760796.480353:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28610560 chunk: [28610560, 28614656) 28614656, 4096 00020000:00200000:3.0:1789760796.480358:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.480376:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6985, 6985] 00000008:00010000:3.0:1789760796.480833:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.480841:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.480855:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28610560, 28614656) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.480869:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28610560=0x1b49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.480938:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.480941:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28614656 00000020:00010000:3.0:1789760796.480949:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.480953:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.480956:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.480959:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.480963:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.481032:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.481041:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.481059:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760796.481160:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148380 - object 0x240000400:5 offset 28610560<>28614656 00000100:00100000:3.0:1789760796.481203:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.481222:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493880576:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.518727:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974303 gen 3 00000100:00100000:2.0:1789760796.518772:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645148380 x1876699493880576/t4294974304(4294974304) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.518823:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760796.518830:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.518839:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.518865:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760796.518888:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493880576:192.168.203.160@tcp:4:dd.0 00000080:00200000:1.0:1789760796.518909:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6985, 6985] 00000080:00200000:1.0:1789760796.518936:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760796.518942:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760796.518957:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28614656 i_blocks 57353 00000020:00200000:1.0:1789760796.518961:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760796.518970:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760796.519072:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28614656, count: 4096 00000080:00200000:1.0:1789760796.519079:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28614656, bytes: 4096 00000080:00200000:1.0:1789760796.519092:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760796.519100:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760796.519102:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28614656, 28618752) 0 0 00000080:00200000:1.0:1789760796.519107:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760796.519122:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6986, 6986] 00020000:00200000:1.0:1789760796.519131:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28614656 chunk: [28614656, 28618752) 28618752, 4096 00020000:00200000:1.0:1789760796.519135:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760796.519154:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6986, 6986] 00000008:00010000:1.0:1789760796.519171:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760796.519178:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb6455ef630, osc lock ffff8eb646c6ad10, flags 0x0 00000080:00200000:1.0:1789760796.519191:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28614656, 28618752) append 0 append lockpos 0 00000080:00200000:1.0:1789760796.519204:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28614656=0x1b4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760796.519549:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760796.519552:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28618752 00000020:00010000:1.0:1789760796.519562:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:1.0:1789760796.519566:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.519569:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb6455ef630 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760796.519572:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760796.519575:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb6455ef630 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760796.519586:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760796.519592:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760796.519608:0:367:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb646e92780 00000008:00100000:1.0:1789760796.520071:0:367:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb65016ea00 - object 0x240000400:5 offset 28614656<>28618752 00000100:00100000:0.0:1789760796.520127:0:2367:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760796.520203:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493880704:192.168.203.160@tcp:4:dd.0 00000100:00100000:0.0:1789760796.554315:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974304 gen 3 00000100:00100000:0.0:1789760796.554360:0:2367:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb65016ea00 x1876699493880704/t4294974305(4294974305) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760796.554452:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb646e92780 00000020:00200000:0.0:1789760796.554460:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760796.554475:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760796.554499:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb646e92780 00000100:00100000:0.0:1789760796.554525:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493880704:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760796.554702:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6986, 6986] 00000080:00200000:2.0:1789760796.554719:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760796.554725:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760796.554741:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28618752 i_blocks 57353 00000020:00200000:2.0:1789760796.554745:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760796.554755:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760796.554790:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28618752, count: 4096 00000080:00200000:2.0:1789760796.554798:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28618752, bytes: 4096 00000080:00200000:2.0:1789760796.554808:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760796.554817:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760796.554819:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28618752, 28622848) 0 0 00000080:00200000:2.0:1789760796.554824:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760796.554841:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6987, 6987] 00020000:00200000:2.0:1789760796.554849:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28618752 chunk: [28618752, 28622848) 28622848, 4096 00020000:00200000:2.0:1789760796.554853:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760796.554873:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6987, 6987] 00000008:00010000:2.0:1789760796.554888:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760796.554895:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760796.554908:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28618752, 28622848) append 0 append lockpos 0 00000080:00200000:2.0:1789760796.555668:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28618752=0x1b4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760796.555743:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760796.555746:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28622848 00000020:00010000:2.0:1789760796.555757:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760796.555760:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.555763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760796.555766:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760796.555770:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760796.555780:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760796.555787:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760796.555807:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760796.557849:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28618752<>28622848 00000100:00100000:3.0:1789760796.557917:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.560313:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880832:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.619848:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974305 gen 3 00000100:00100000:2.0:1789760796.619897:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb650f81180 x1876699493880832/t4294974306(4294974306) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.619962:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:2.0:1789760796.619969:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.620097:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.620118:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:2.0:1789760796.620143:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493880832:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.620586:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6987, 6987] 00000080:00200000:3.0:1789760796.620604:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.620611:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.620628:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28622848 i_blocks 57353 00000020:00200000:3.0:1789760796.620633:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.620644:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.620678:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28622848, count: 4096 00000080:00200000:3.0:1789760796.620685:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28622848, bytes: 4096 00000080:00200000:3.0:1789760796.620697:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.620707:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.620709:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28622848, 28626944) 0 0 00000080:00200000:3.0:1789760796.620714:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.620733:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6988, 6988] 00020000:00200000:3.0:1789760796.620743:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28622848 chunk: [28622848, 28626944) 28626944, 4096 00020000:00200000:3.0:1789760796.620747:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.620769:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6988, 6988] 00000008:00010000:3.0:1789760796.620788:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.620794:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.620807:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28622848, 28626944) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.620833:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28622848=0x1b4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.620896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.620900:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28626944 00000020:00010000:3.0:1789760796.621036:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.621042:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.621045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.621049:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.621052:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.621061:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.621069:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.621087:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760796.621195:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514ad80 - object 0x240000400:5 offset 28622848<>28626944 00000100:00100000:3.0:1789760796.621246:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.621267:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493880960:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.657486:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974307 gen 3 00000008:00010000:2.0:1789760796.657548:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760796.657556:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.657566:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.657580:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760796.657610:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493880960:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.657791:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6988, 6988] 00000080:00200000:3.0:1789760796.657807:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.657813:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.657828:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28626944 i_blocks 57353 00000020:00200000:3.0:1789760796.657832:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.657841:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.657875:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28626944, count: 4096 00000080:00200000:3.0:1789760796.657881:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28626944, bytes: 4096 00000080:00200000:3.0:1789760796.657889:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.657896:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.657898:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28626944, 28631040) 0 0 00000080:00200000:3.0:1789760796.657901:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.657925:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6989, 6989] 00020000:00200000:3.0:1789760796.657933:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28626944 chunk: [28626944, 28631040) 28631040, 4096 00020000:00200000:3.0:1789760796.657938:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.657957:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6989, 6989] 00000008:00010000:3.0:1789760796.657977:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.657983:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.657996:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28626944, 28631040) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.658012:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28626944=0x1b4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.658068:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.658071:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28631040 00000020:00010000:3.0:1789760796.658079:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.658083:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.658086:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.658090:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.658093:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.658101:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.658108:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.658124:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760796.658220:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b480 - object 0x240000400:5 offset 28626944<>28631040 00000100:00100000:3.0:1789760796.658584:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881088:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760796.690891:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760796.690907:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.690927:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.690947:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000080:00200000:3.0:1789760796.691107:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6989, 6989] 00000080:00200000:3.0:1789760796.691126:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.691132:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.691148:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28631040 i_blocks 57353 00000020:00200000:3.0:1789760796.691151:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.691160:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.691195:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28631040, count: 4096 00000080:00200000:3.0:1789760796.691203:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28631040, bytes: 4096 00000080:00200000:3.0:1789760796.691212:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.691221:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.691223:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28631040, 28635136) 0 0 00000080:00200000:3.0:1789760796.691228:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.691243:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6990, 6990] 00020000:00200000:3.0:1789760796.691252:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28631040 chunk: [28631040, 28635136) 28635136, 4096 00020000:00200000:3.0:1789760796.691257:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.691274:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6990, 6990] 00000008:00010000:3.0:1789760796.691294:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.691300:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.691312:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28631040, 28635136) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.691325:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28631040=0x1b4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.691369:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.691371:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28635136 00000020:00010000:3.0:1789760796.691379:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.691382:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.691385:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.691388:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.691477:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.691487:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.691493:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760796.691597:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881088:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760796.691626:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:2.0:1789760796.691725:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28631040<>28635136 00000100:00100000:2.0:1789760796.691764:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760796.691780:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881216:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.731697:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974309 gen 3 00000008:00010000:2.0:1789760796.731770:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760796.731779:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.732027:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.732117:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760796.732151:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881216:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.732239:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6990, 6990] 00000080:00200000:3.0:1789760796.732257:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.732263:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.732278:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28635136 i_blocks 57353 00000020:00200000:3.0:1789760796.732281:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.732291:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.732324:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28635136, count: 4096 00000080:00200000:3.0:1789760796.732332:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28635136, bytes: 4096 00000080:00200000:3.0:1789760796.732346:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.732356:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.732358:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28635136, 28639232) 0 0 00000080:00200000:3.0:1789760796.732363:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.732379:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6991, 6991] 00020000:00200000:3.0:1789760796.732388:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28635136 chunk: [28635136, 28639232) 28639232, 4096 00020000:00200000:3.0:1789760796.732600:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.732626:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6991, 6991] 00000008:00010000:3.0:1789760796.732646:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.732652:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.732665:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28635136, 28639232) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.732683:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28635136=0x1b4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.732749:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.732752:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28639232 00000020:00010000:3.0:1789760796.732761:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.732765:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.732768:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.732772:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.732775:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.732782:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.733726:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.733753:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760796.734248:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148000 - object 0x240000400:5 offset 28635136<>28639232 00000100:00100000:3.0:1789760796.734312:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.734338:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881344:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.785164:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974310 gen 3 00000008:00010000:2.0:1789760796.785221:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760796.785230:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.785244:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.785264:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760796.785293:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881344:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.785352:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6991, 6991] 00000080:00200000:3.0:1789760796.785369:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.785376:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.785491:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28639232 i_blocks 57353 00000020:00200000:3.0:1789760796.785496:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.785507:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.785542:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28639232, count: 4096 00000080:00200000:3.0:1789760796.785551:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28639232, bytes: 4096 00000080:00200000:3.0:1789760796.785561:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.785570:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.785572:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28639232, 28643328) 0 0 00000080:00200000:3.0:1789760796.785577:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.785596:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6992, 6992] 00020000:00200000:3.0:1789760796.785604:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28639232 chunk: [28639232, 28643328) 28643328, 4096 00020000:00200000:3.0:1789760796.785609:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.785628:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6992, 6992] 00000008:00010000:3.0:1789760796.785647:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.785652:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.785665:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28639232, 28643328) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.785679:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28639232=0x1b50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.785739:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.785742:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28643328 00000020:00010000:3.0:1789760796.785750:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.785754:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.786237:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.786242:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.786245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.786255:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.786263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.786282:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760796.786384:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149180 - object 0x240000400:5 offset 28639232<>28643328 00000100:00100000:2.0:1789760796.786690:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881472:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760796.833619:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974311 gen 3 00000008:00010000:2.0:1789760796.833670:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760796.833678:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.833690:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.833711:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760796.833739:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881472:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.834884:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6992, 6992] 00000080:00200000:3.0:1789760796.834906:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.834913:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.834937:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28643328 i_blocks 57353 00000020:00200000:3.0:1789760796.834943:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.834955:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.834991:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28643328, count: 4096 00000080:00200000:3.0:1789760796.834999:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28643328, bytes: 4096 00000080:00200000:3.0:1789760796.835012:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.835021:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.835023:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28643328, 28647424) 0 0 00000080:00200000:3.0:1789760796.835029:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.835045:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6993, 6993] 00020000:00200000:3.0:1789760796.835053:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28643328 chunk: [28643328, 28647424) 28647424, 4096 00020000:00200000:3.0:1789760796.835058:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.835080:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6993, 6993] 00000008:00010000:3.0:1789760796.835100:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.835107:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.835119:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28643328, 28647424) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.835133:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28643328=0x1b51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.835200:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.835203:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28647424 00000020:00010000:3.0:1789760796.835211:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.835214:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.835217:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.835221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.835225:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.835233:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.835241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.835258:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760796.835363:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b800 - object 0x240000400:5 offset 28643328<>28647424 00000100:00100000:2.0:1789760796.835581:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881600:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760796.882826:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760796.882841:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.882854:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.882873:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760796.883048:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881600:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.883139:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6993, 6993] 00000080:00200000:3.0:1789760796.883154:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.883160:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.883176:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28647424 i_blocks 57353 00000020:00200000:3.0:1789760796.883180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.883189:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.883225:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28647424, count: 4096 00000080:00200000:3.0:1789760796.883232:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28647424, bytes: 4096 00000080:00200000:3.0:1789760796.883243:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.883252:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.883254:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28647424, 28651520) 0 0 00000080:00200000:3.0:1789760796.883258:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.883277:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6994, 6994] 00020000:00200000:3.0:1789760796.883285:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28647424 chunk: [28647424, 28651520) 28651520, 4096 00020000:00200000:3.0:1789760796.883289:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.883310:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6994, 6994] 00000008:00010000:3.0:1789760796.883327:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.883333:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.883346:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28647424, 28651520) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.883362:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28647424=0x1b52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.883443:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.883447:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28651520 00000020:00010000:3.0:1789760796.883456:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.883460:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.883463:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.883466:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.883469:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.883476:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.883483:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.883499:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760796.884134:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 28647424<>28651520 00000100:00100000:2.0:1789760796.888058:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881728:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760796.909519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760796.953206:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974312 gen 3 00000100:00100000:2.0:1789760796.953251:0:2369:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514aa00 x1876699493881728/t4294974313(4294974313) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760796.953303:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760796.953311:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.953321:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.953344:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760796.953370:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493881728:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760796.953745:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6994, 6994] 00000080:00200000:3.0:1789760796.953762:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.953768:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.953793:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28651520 i_blocks 57353 00000020:00200000:3.0:1789760796.953797:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.953807:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.953842:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28651520, count: 4096 00000080:00200000:3.0:1789760796.953850:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28651520, bytes: 4096 00000080:00200000:3.0:1789760796.953861:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.953871:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.953873:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28651520, 28655616) 0 0 00000080:00200000:3.0:1789760796.953877:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.954096:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6995, 6995] 00020000:00200000:3.0:1789760796.954105:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28651520 chunk: [28651520, 28655616) 28655616, 4096 00020000:00200000:3.0:1789760796.954110:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.954132:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6995, 6995] 00000008:00010000:3.0:1789760796.954151:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.954157:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760796.954169:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28651520, 28655616) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.954183:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28651520=0x1b53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.954248:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.954251:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28655616 00000020:00010000:3.0:1789760796.954259:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.954263:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.954266:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.954269:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.954272:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.954280:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.954288:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.954307:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760796.954458:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 28651520<>28655616 00000100:00100000:2.0:1789760796.955697:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881856:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.006187:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974313 gen 3 00000100:00100000:2.0:1789760797.006233:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb645149500 x1876699493881856/t4294974314(4294974314) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760812 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760797.006284:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:2.0:1789760797.006292:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.006301:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.006323:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:2.0:1789760797.006347:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881856:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.006623:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6995, 6995] 00000080:00200000:3.0:1789760797.006639:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.006646:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.006661:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28655616 i_blocks 57353 00000020:00200000:3.0:1789760797.006665:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.006674:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.006708:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28655616, count: 4096 00000080:00200000:3.0:1789760797.006715:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28655616, bytes: 4096 00000080:00200000:3.0:1789760797.006726:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.006735:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.006737:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28655616, 28659712) 0 0 00000080:00200000:3.0:1789760797.006741:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.006757:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6996, 6996] 00020000:00200000:3.0:1789760797.006766:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28655616 chunk: [28655616, 28659712) 28659712, 4096 00020000:00200000:3.0:1789760797.006772:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.006792:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6996, 6996] 00000008:00010000:3.0:1789760797.006811:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.006818:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.006831:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28655616, 28659712) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.006848:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28655616=0x1b54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.008600:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.008609:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28659712 00000020:00010000:3.0:1789760797.008623:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.008628:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.008632:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.008636:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.008639:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.008652:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.008661:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.008692:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760797.008815:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 28655616<>28659712 00000100:00100000:3.0:1789760797.008871:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.008897:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881984:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.061626:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974315 gen 3 00000008:00010000:2.0:1789760797.061691:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760797.061700:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.061709:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.061729:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760797.061760:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493881984:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.062008:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6996, 6996] 00000080:00200000:3.0:1789760797.062032:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.062038:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.062052:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28659712 i_blocks 57353 00000020:00200000:3.0:1789760797.062055:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.062065:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.062671:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28659712, count: 4096 00000080:00200000:3.0:1789760797.062682:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28659712, bytes: 4096 00000080:00200000:3.0:1789760797.062694:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.062704:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.062706:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28659712, 28663808) 0 0 00000080:00200000:3.0:1789760797.062712:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.062729:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6997, 6997] 00020000:00200000:3.0:1789760797.062738:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28659712 chunk: [28659712, 28663808) 28663808, 4096 00020000:00200000:3.0:1789760797.062744:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.062764:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6997, 6997] 00000008:00010000:3.0:1789760797.062784:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.062790:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.062803:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28659712, 28663808) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.062818:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28659712=0x1b55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.062881:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.062884:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28663808 00000020:00010000:3.0:1789760797.062893:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.062897:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.062899:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.062922:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.062926:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.062935:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.062942:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.062963:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760797.063067:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 28659712<>28663808 00000100:00100000:2.0:1789760797.063300:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.063324:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882112:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.105796:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974316 gen 3 00000008:00010000:2.0:1789760797.105850:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760797.105859:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.105870:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.105894:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760797.105943:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882112:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.106221:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6997, 6997] 00000080:00200000:3.0:1789760797.106238:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.106245:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.106261:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28663808 i_blocks 57353 00000020:00200000:3.0:1789760797.106265:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.106276:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.106312:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28663808, count: 4096 00000080:00200000:3.0:1789760797.106319:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28663808, bytes: 4096 00000080:00200000:3.0:1789760797.106330:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.106339:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.106341:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28663808, 28667904) 0 0 00000080:00200000:3.0:1789760797.106347:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.106363:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6998, 6998] 00020000:00200000:3.0:1789760797.106371:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28663808 chunk: [28663808, 28667904) 28667904, 4096 00020000:00200000:3.0:1789760797.106375:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.106490:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6998, 6998] 00000008:00010000:3.0:1789760797.106511:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.106518:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.106532:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28663808, 28667904) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.106547:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28663808=0x1b56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.106672:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.106675:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28667904 00000020:00010000:3.0:1789760797.106684:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.106687:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.106690:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.106694:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.106698:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.106705:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.106712:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.106729:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760797.106833:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 28663808<>28667904 00000100:00100000:2.0:1789760797.107183:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760797.107206:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882240:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.150656:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974317 gen 3 00000008:00010000:2.0:1789760797.150704:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760797.150712:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.150723:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.150743:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760797.150772:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882240:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.150874:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6998, 6998] 00000080:00200000:3.0:1789760797.150893:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.150900:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.150926:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28667904 i_blocks 57353 00000020:00200000:3.0:1789760797.150930:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.150950:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.150992:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28667904, count: 4096 00000080:00200000:3.0:1789760797.150999:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28667904, bytes: 4096 00000080:00200000:3.0:1789760797.151008:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.151025:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.151027:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28667904, 28672000) 0 0 00000080:00200000:3.0:1789760797.151033:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.151048:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [6999, 6999] 00020000:00200000:3.0:1789760797.151057:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28667904 chunk: [28667904, 28672000) 28672000, 4096 00020000:00200000:3.0:1789760797.151061:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.151080:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6999, 6999] 00000008:00010000:3.0:1789760797.151098:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.151114:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.151128:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28667904, 28672000) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.151141:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28667904=0x1b57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.151198:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.151201:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28672000 00000020:00010000:3.0:1789760797.151209:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.151213:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.151216:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.151219:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.151222:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.151230:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.151236:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.151251:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760797.151344:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 28667904<>28672000 00000100:00100000:2.0:1789760797.151622:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.151642:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882368:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.209440:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974318 gen 3 00000008:00010000:2.0:1789760797.209552:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760797.209561:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.209572:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.209647:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760797.209674:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882368:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.209754:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [6999, 6999] 00000080:00200000:3.0:1789760797.209769:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.209775:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.209791:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28672000 i_blocks 57353 00000020:00200000:3.0:1789760797.209794:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.209803:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.209833:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28672000, count: 4096 00000080:00200000:3.0:1789760797.209840:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28672000, bytes: 4096 00000080:00200000:3.0:1789760797.209850:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.209858:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.209861:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28672000, 28676096) 0 0 00000080:00200000:3.0:1789760797.209866:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.209882:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7000, 7000] 00020000:00200000:3.0:1789760797.209890:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28672000 chunk: [28672000, 28676096) 28676096, 4096 00020000:00200000:3.0:1789760797.209894:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.209924:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7000, 7000] 00000008:00010000:3.0:1789760797.209944:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.209951:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.209962:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28672000, 28676096) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.209975:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28672000=0x1b58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.210089:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.210092:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28676096 00000020:00010000:3.0:1789760797.210100:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.210104:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.210107:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.210111:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.210114:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.210121:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.210127:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.210141:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760797.210240:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 28672000<>28676096 00000100:00100000:2.0:1789760797.210301:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882496:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.257383:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974319 gen 3 00000008:00010000:2.0:1789760797.257469:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760797.257479:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.257490:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.257551:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760797.257580:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493882496:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.257850:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7000, 7000] 00000080:00200000:3.0:1789760797.257870:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.257876:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.257892:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28676096 i_blocks 57353 00000020:00200000:3.0:1789760797.257896:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.257907:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.258087:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28676096, count: 4096 00000080:00200000:3.0:1789760797.258095:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28676096, bytes: 4096 00000080:00200000:3.0:1789760797.258105:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.258115:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.258117:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28676096, 28680192) 0 0 00000080:00200000:3.0:1789760797.258124:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.258142:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7001, 7001] 00020000:00200000:3.0:1789760797.258150:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28676096 chunk: [28676096, 28680192) 28680192, 4096 00020000:00200000:3.0:1789760797.258155:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.258173:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7001, 7001] 00000008:00010000:3.0:1789760797.258204:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.258211:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.258222:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28676096, 28680192) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.258237:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28676096=0x1b59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.258555:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.258559:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28680192 00000020:00010000:3.0:1789760797.258570:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.258573:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.258576:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.258580:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.258583:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.258593:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.258600:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.258619:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760797.258728:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645148a80 - object 0x240000400:5 offset 28676096<>28680192 00000100:00100000:2.0:1789760797.258857:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882624:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760797.320064:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760797.320080:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.320091:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.320240:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760797.320271:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882624:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.321055:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7001, 7001] 00000080:00200000:3.0:1789760797.321074:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.321080:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.321097:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28680192 i_blocks 57353 00000020:00200000:3.0:1789760797.321101:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.321112:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.321146:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28680192, count: 4096 00000080:00200000:3.0:1789760797.321154:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28680192, bytes: 4096 00000080:00200000:3.0:1789760797.321166:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.321175:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.321177:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28680192, 28684288) 0 0 00000080:00200000:3.0:1789760797.321183:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.321201:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7002, 7002] 00020000:00200000:3.0:1789760797.321210:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28680192 chunk: [28680192, 28684288) 28684288, 4096 00020000:00200000:3.0:1789760797.321214:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.321234:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7002, 7002] 00000008:00010000:3.0:1789760797.321254:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.321260:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.321273:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28680192, 28684288) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.321287:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28680192=0x1b5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.321349:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.321352:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28684288 00000020:00010000:3.0:1789760797.321360:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.321364:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.321367:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.321371:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.321374:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.321381:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.321388:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.321628:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27f00 00000008:00100000:3.0:1789760797.321736:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a680 - object 0x240000400:5 offset 28680192<>28684288 00000100:00100000:2.0:1789760797.321792:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760797.322058:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882752:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.378333:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974320 gen 3 00000100:00100000:2.0:1789760797.378377:0:2370:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8eb64514a680 x1876699493882752/t4294974321(4294974321) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760813 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760797.378612:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27f00 00000020:00200000:2.0:1789760797.378620:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.378765:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.378903:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27f00 00000100:00100000:2.0:1789760797.378943:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493882752:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760797.379544:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7002, 7002] 00000080:00200000:0.0:1789760797.379562:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.379569:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.379586:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28684288 i_blocks 57353 00000020:00200000:0.0:1789760797.379590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.379601:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.379638:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28684288, count: 4096 00000080:00200000:0.0:1789760797.379646:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28684288, bytes: 4096 00000080:00200000:0.0:1789760797.379661:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.379670:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.379673:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28684288, 28688384) 0 0 00000080:00200000:0.0:1789760797.379678:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.379695:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7003, 7003] 00020000:00200000:0.0:1789760797.379704:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28684288 chunk: [28684288, 28688384) 28688384, 4096 00020000:00200000:0.0:1789760797.379709:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.379730:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7003, 7003] 00000008:00010000:0.0:1789760797.379751:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.379757:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760797.379770:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28684288, 28688384) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.380051:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28684288=0x1b5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.380128:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.380132:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28688384 00000020:00010000:0.0:1789760797.380141:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.380145:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.380149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.380152:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.380156:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.380165:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.380173:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.380195:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb780 00000008:00100000:0.0:1789760797.380308:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304d180 - object 0x240000400:5 offset 28684288<>28688384 00000100:00100000:1.0:1789760797.380738:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493882880:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760797.431138:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974322 gen 3 00000008:00010000:1.0:1789760797.431202:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb780 00000020:00200000:1.0:1789760797.431210:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.431222:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.431243:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb780 00000100:00100000:1.0:1789760797.431272:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493882880:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.431566:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7003, 7003] 00000080:00200000:3.0:1789760797.431583:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.431588:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.431605:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28688384 i_blocks 57353 00000020:00200000:3.0:1789760797.431609:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.431620:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.431655:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28688384, count: 4096 00000080:00200000:3.0:1789760797.431662:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28688384, bytes: 4096 00000080:00200000:3.0:1789760797.431672:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.431682:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.431684:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28688384, 28692480) 0 0 00000080:00200000:3.0:1789760797.431690:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.431706:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7004, 7004] 00020000:00200000:3.0:1789760797.431714:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28688384 chunk: [28688384, 28692480) 28692480, 4096 00020000:00200000:3.0:1789760797.431719:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.431740:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7004, 7004] 00000008:00010000:3.0:1789760797.431759:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.431765:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.431778:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28688384, 28692480) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.431792:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28688384=0x1b5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.431853:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.431856:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28692480 00000020:00010000:3.0:1789760797.431866:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.431870:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.431873:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.431876:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.431880:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.431889:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.431895:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.431923:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d273c0 00000008:00100000:3.0:1789760797.432026:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149c00 - object 0x240000400:5 offset 28688384<>28692480 00000100:00100000:2.0:1789760797.432110:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493883008:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.491246:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974323 gen 3 00000008:00010000:2.0:1789760797.491296:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d273c0 00000020:00200000:2.0:1789760797.491305:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.491315:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.491335:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d273c0 00000100:00100000:2.0:1789760797.491363:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493883008:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.491547:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7004, 7004] 00000080:00200000:3.0:1789760797.491564:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.491570:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.491586:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28692480 i_blocks 57353 00000020:00200000:3.0:1789760797.491590:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.491600:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.491637:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28692480, count: 4096 00000080:00200000:3.0:1789760797.491644:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28692480, bytes: 4096 00000080:00200000:3.0:1789760797.491653:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.491661:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.491663:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28692480, 28696576) 0 0 00000080:00200000:3.0:1789760797.491669:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.491686:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7005, 7005] 00020000:00200000:3.0:1789760797.491695:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28692480 chunk: [28692480, 28696576) 28696576, 4096 00020000:00200000:3.0:1789760797.491699:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.491718:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7005, 7005] 00000008:00010000:3.0:1789760797.491737:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.491743:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.491755:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28692480, 28696576) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.491770:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28692480=0x1b5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.491830:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.491833:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28696576 00000020:00010000:3.0:1789760797.491842:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.491845:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.491848:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.491851:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.491854:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.491863:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.491871:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.491889:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27000 00000008:00100000:3.0:1789760797.492003:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514b100 - object 0x240000400:5 offset 28692480<>28696576 00000100:00100000:2.0:1789760797.492192:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883136:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.554535:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974324 gen 3 00000008:00010000:2.0:1789760797.554585:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27000 00000020:00200000:2.0:1789760797.554592:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.554602:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.554621:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27000 00000100:00100000:2.0:1789760797.554648:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883136:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.554739:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7005, 7005] 00000080:00200000:3.0:1789760797.554756:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.554762:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.554776:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28696576 i_blocks 57353 00000020:00200000:3.0:1789760797.554780:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.554789:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.554829:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28696576, count: 4096 00000080:00200000:3.0:1789760797.554836:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28696576, bytes: 4096 00000080:00200000:3.0:1789760797.554845:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.554853:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.554856:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28696576, 28700672) 0 0 00000080:00200000:3.0:1789760797.554861:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.554876:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7006, 7006] 00020000:00200000:3.0:1789760797.554884:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28696576 chunk: [28696576, 28700672) 28700672, 4096 00020000:00200000:3.0:1789760797.554889:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.554907:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7006, 7006] 00000008:00010000:3.0:1789760797.554931:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.554937:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.554949:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28696576, 28700672) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.554971:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28696576=0x1b5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.555030:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.555033:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28700672 00000020:00010000:3.0:1789760797.555041:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.555045:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.555048:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.555051:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.555054:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.555061:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.555067:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.555081:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d276c0 00000008:00100000:3.0:1789760797.555185:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149880 - object 0x240000400:5 offset 28696576<>28700672 00000100:00100000:2.0:1789760797.556251:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.556279:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883264:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.588340:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974325 gen 3 00000008:00010000:2.0:1789760797.588483:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d276c0 00000020:00200000:2.0:1789760797.588493:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.588505:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.588525:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d276c0 00000100:00100000:2.0:1789760797.588556:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883264:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.588671:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7006, 7006] 00000080:00200000:3.0:1789760797.588689:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.588696:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.588713:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28700672 i_blocks 57353 00000020:00200000:3.0:1789760797.588717:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.588728:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.588764:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28700672, count: 4096 00000080:00200000:3.0:1789760797.588773:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28700672, bytes: 4096 00000080:00200000:3.0:1789760797.588784:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.588793:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.588796:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28700672, 28704768) 0 0 00000080:00200000:3.0:1789760797.588802:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.588893:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7007, 7007] 00020000:00200000:3.0:1789760797.588904:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28700672 chunk: [28700672, 28704768) 28704768, 4096 00020000:00200000:3.0:1789760797.588909:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.588941:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7007, 7007] 00000008:00010000:3.0:1789760797.588961:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.589030:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.589044:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28700672, 28704768) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.589059:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28700672=0x1b5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.589123:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.589126:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28704768 00000020:00010000:3.0:1789760797.589135:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.589139:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.589142:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.589146:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.589149:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.589158:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.589164:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.589182:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27c00 00000008:00100000:3.0:1789760797.589292:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514a300 - object 0x240000400:5 offset 28700672<>28704768 00000100:00100000:2.0:1789760797.589365:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883392:192.168.203.160@tcp:4:dd.0 00000100:00100000:2.0:1789760797.629878:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974326 gen 3 00000008:00010000:2.0:1789760797.629942:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27c00 00000020:00200000:2.0:1789760797.629951:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.629962:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.629987:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27c00 00000100:00100000:2.0:1789760797.630016:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883392:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.630126:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7007, 7007] 00000080:00200000:3.0:1789760797.630144:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.630150:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.630166:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28704768 i_blocks 57353 00000020:00200000:3.0:1789760797.630170:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.630180:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.630217:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28704768, count: 4096 00000080:00200000:3.0:1789760797.630225:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28704768, bytes: 4096 00000080:00200000:3.0:1789760797.630235:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.630243:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.630245:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28704768, 28708864) 0 0 00000080:00200000:3.0:1789760797.630250:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.630266:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7008, 7008] 00020000:00200000:3.0:1789760797.630274:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28704768 chunk: [28704768, 28708864) 28708864, 4096 00020000:00200000:3.0:1789760797.630279:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.630298:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7008, 7008] 00000008:00010000:3.0:1789760797.630317:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.630322:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.630334:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28704768, 28708864) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.630349:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28704768=0x1b60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.630716:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.630719:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28708864 00000020:00010000:3.0:1789760797.630728:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.630732:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.630735:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.630739:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.630742:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.630751:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.630757:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.630792:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27240 00000008:00100000:3.0:1789760797.630903:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb645149500 - object 0x240000400:5 offset 28704768<>28708864 00000100:00100000:3.0:1789760797.630951:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493883520:192.168.203.160@tcp:4:dd.0 00000008:00010000:2.0:1789760797.660289:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27240 00000020:00200000:2.0:1789760797.660306:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.660317:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.660425:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27240 00000100:00100000:2.0:1789760797.660457:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493883520:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760797.660767:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7008, 7008] 00000080:00200000:0.0:1789760797.660838:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.660843:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.660857:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28708864 i_blocks 57353 00000020:00200000:0.0:1789760797.660861:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.660871:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.660904:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28708864, count: 4096 00000080:00200000:0.0:1789760797.660922:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28708864, bytes: 4096 00000080:00200000:0.0:1789760797.660935:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.660944:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.660946:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28708864, 28712960) 0 0 00000080:00200000:0.0:1789760797.660950:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.660966:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7009, 7009] 00020000:00200000:0.0:1789760797.660974:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28708864 chunk: [28708864, 28712960) 28712960, 4096 00020000:00200000:0.0:1789760797.660979:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.660998:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7009, 7009] 00000008:00010000:0.0:1789760797.661118:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.661124:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760797.661135:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28708864, 28712960) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.661148:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28708864=0x1b61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.661203:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.661207:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28712960 00000020:00010000:0.0:1789760797.661215:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.661218:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.661221:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.661224:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.661227:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.661234:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.661241:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.661258:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000008:00100000:0.0:1789760797.661355:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c000 - object 0x240000400:5 offset 28708864<>28712960 00000100:00100000:1.0:1789760797.661446:0:2368:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493883648:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760797.694273:0:2368:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974328 gen 3 00000008:00010000:1.0:1789760797.694343:0:2368:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000020:00200000:1.0:1789760797.694352:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.694362:0:2368:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.694386:0:2368:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbb3c0 00000100:00100000:1.0:1789760797.694518:0:2368:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2368:1876699493883648:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.694692:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7009, 7009] 00000080:00200000:3.0:1789760797.694709:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.694714:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.694730:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28712960 i_blocks 57353 00000020:00200000:3.0:1789760797.694734:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.694744:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.694777:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28712960, count: 4096 00000080:00200000:3.0:1789760797.694785:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28712960, bytes: 4096 00000080:00200000:3.0:1789760797.694796:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.694806:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.694808:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28712960, 28717056) 0 0 00000080:00200000:3.0:1789760797.694813:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.694884:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7010, 7010] 00020000:00200000:3.0:1789760797.694892:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28712960 chunk: [28712960, 28717056) 28717056, 4096 00020000:00200000:3.0:1789760797.694897:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.694926:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7010, 7010] 00000008:00010000:3.0:1789760797.694946:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.694952:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.694963:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28712960, 28717056) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.694977:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28712960=0x1b62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.695222:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.695226:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28717056 00000020:00010000:3.0:1789760797.695235:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.695238:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.695241:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.695245:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.695248:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.695257:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.695263:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.695280:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760797.695384:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 28712960<>28717056 00000100:00100000:2.0:1789760797.696154:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883776:192.168.203.160@tcp:4:dd.0 00000008:00010000:3.0:1789760797.733919:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb649d27900 00000020:00200000:3.0:1789760797.733937:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.733949:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.733971:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb649d27900 00000100:00100000:3.0:1789760797.734004:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883776:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760797.734290:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7010, 7010] 00000080:00200000:2.0:1789760797.734307:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.734313:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.734330:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28717056 i_blocks 57353 00000020:00200000:2.0:1789760797.734334:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.734345:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.734384:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28717056, count: 4096 00000080:00200000:2.0:1789760797.734440:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28717056, bytes: 4096 00000080:00200000:2.0:1789760797.734454:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.734463:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.734466:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28717056, 28721152) 0 0 00000080:00200000:2.0:1789760797.734471:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.734489:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7011, 7011] 00020000:00200000:2.0:1789760797.734498:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28717056 chunk: [28717056, 28721152) 28721152, 4096 00020000:00200000:2.0:1789760797.734503:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.734523:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7011, 7011] 00000008:00010000:2.0:1789760797.735029:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.735036:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760797.735050:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28717056, 28721152) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.735065:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28717056=0x1b63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.735138:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.735142:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28721152 00000020:00010000:2.0:1789760797.735150:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.735154:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.735157:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.735161:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.735164:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.735172:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.735180:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.735201:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3be40 00000008:00100000:2.0:1789760797.735310:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f81180 - object 0x240000400:5 offset 28717056<>28721152 00000100:00100000:3.0:1789760797.735427:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.735450:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883904:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760797.779292:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974330 gen 3 00000008:00010000:3.0:1789760797.779369:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3be40 00000020:00200000:3.0:1789760797.779380:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.779837:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.779903:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3be40 00000100:00100000:3.0:1789760797.779944:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493883904:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760797.780025:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7011, 7011] 00000080:00200000:2.0:1789760797.780041:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.780048:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.780064:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28721152 i_blocks 57353 00000020:00200000:2.0:1789760797.780068:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.780079:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.780114:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28721152, count: 4096 00000080:00200000:2.0:1789760797.780121:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28721152, bytes: 4096 00000080:00200000:2.0:1789760797.780133:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.780142:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.780145:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28721152, 28725248) 0 0 00000080:00200000:2.0:1789760797.780149:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.780165:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7012, 7012] 00020000:00200000:2.0:1789760797.780173:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28721152 chunk: [28721152, 28725248) 28725248, 4096 00020000:00200000:2.0:1789760797.780179:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.780197:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7012, 7012] 00000008:00010000:2.0:1789760797.780213:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.780219:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760797.780303:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28721152, 28725248) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.780319:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28721152=0x1b64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.780380:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.780841:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28725248 00000020:00010000:2.0:1789760797.780853:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.780857:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.780861:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.780864:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.780867:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.780876:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.780884:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.780902:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b180 00000008:00100000:2.0:1789760797.781019:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f83480 - object 0x240000400:5 offset 28721152<>28725248 00000100:00100000:3.0:1789760797.781088:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493884032:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760797.818168:0:2369:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974331 gen 3 00000008:00010000:3.0:1789760797.818222:0:2369:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b180 00000020:00200000:3.0:1789760797.818241:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.818252:0:2369:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.818272:0:2369:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b180 00000100:00100000:3.0:1789760797.818303:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493884032:192.168.203.160@tcp:4:dd.0 00000080:00200000:2.0:1789760797.818966:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7012, 7012] 00000080:00200000:2.0:1789760797.818984:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.818990:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.819005:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28725248 i_blocks 57353 00000020:00200000:2.0:1789760797.819010:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.819021:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.819055:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28725248, count: 4096 00000080:00200000:2.0:1789760797.819062:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28725248, bytes: 4096 00000080:00200000:2.0:1789760797.819073:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.819081:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.819083:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28725248, 28729344) 0 0 00000080:00200000:2.0:1789760797.819088:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.819103:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7013, 7013] 00020000:00200000:2.0:1789760797.819111:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28725248 chunk: [28725248, 28729344) 28729344, 4096 00020000:00200000:2.0:1789760797.819116:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.819137:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7013, 7013] 00000008:00010000:2.0:1789760797.819165:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.819172:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64750f930, osc lock ffff8eb650f44d10, flags 0x0 00000080:00200000:2.0:1789760797.819184:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28725248, 28729344) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.819199:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28725248=0x1b65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.819267:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.819271:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28729344 00000020:00010000:2.0:1789760797.819279:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.819283:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.819286:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64750f930 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.819290:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.819293:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64750f930 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.819302:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.819310:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.819328:0:9897:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb644e3b780 00000008:00100000:2.0:1789760797.821355:0:9897:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb650f80e00 - object 0x240000400:5 offset 28725248<>28729344 00000100:00100000:2.0:1789760797.822654:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493884160:192.168.203.160@tcp:4:dd.0 00000100:00100000:3.0:1789760797.871978:0:2370:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974332 gen 3 00000008:00010000:3.0:1789760797.872032:0:2370:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb644e3b780 00000020:00200000:3.0:1789760797.872040:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.872052:0:2370:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.872128:0:2370:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb644e3b780 00000100:00100000:3.0:1789760797.872159:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493884160:192.168.203.160@tcp:4:dd.0 00000080:00200000:0.0:1789760797.872250:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7013, 7013] 00000080:00200000:0.0:1789760797.872268:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.872274:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.872289:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28729344 i_blocks 57353 00000020:00200000:0.0:1789760797.872292:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.872304:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.872338:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28729344, count: 4096 00000080:00200000:0.0:1789760797.872345:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28729344, bytes: 4096 00000080:00200000:0.0:1789760797.872356:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.872366:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.872368:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28729344, 28733440) 0 0 00000080:00200000:0.0:1789760797.872373:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.873165:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7014, 7014] 00020000:00200000:0.0:1789760797.873178:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28729344 chunk: [28729344, 28733440) 28733440, 4096 00020000:00200000:0.0:1789760797.873183:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.873206:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7014, 7014] 00000008:00010000:0.0:1789760797.873225:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.873232:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb64508d1b0, osc lock ffff8eb644500130, flags 0x0 00000080:00200000:0.0:1789760797.873243:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28729344, 28733440) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.873257:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28729344=0x1b66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.873316:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.873320:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28733440 00000020:00010000:0.0:1789760797.873328:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.873332:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.873335:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb64508d1b0 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.873339:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.873342:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb64508d1b0 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.873350:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.873358:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.873391:0:11827:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb652bbbd80 00000008:00100000:0.0:1789760797.873525:0:11827:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64304c380 - object 0x240000400:5 offset 28729344<>28733440 00000100:00100000:1.0:1789760797.873606:0:2367:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493884288:192.168.203.160@tcp:4:dd.0 00000100:00100000:1.0:1789760797.922161:0:2367:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8eb6513ab000: committing for last_committed 4294974333 gen 3 00000008:00010000:1.0:1789760797.922211:0:2367:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8eb652bbbd80 00000020:00200000:1.0:1789760797.922221:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.922233:0:2367:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.922320:0:2367:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8eb652bbbd80 00000100:00100000:1.0:1789760797.922352:0:2367:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2367:1876699493884288:192.168.203.160@tcp:4:dd.0 00000080:00200000:3.0:1789760797.923383:0:11778:0:(file.c:2205:ll_file_io_generic()) Range unlock [7014, 7014] 00000080:00200000:3.0:1789760797.923429:0:11778:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760797.923436:0:11778:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760797.923454:0:11778:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28733440 i_blocks 57353 00000020:00200000:3.0:1789760797.923460:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760797.923473:0:11778:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760797.923509:0:11778:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28733440, count: 4096 00000080:00200000:3.0:1789760797.923516:0:11778:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28733440, bytes: 4096 00000080:00200000:3.0:1789760797.923529:0:11778:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760797.923539:0:11778:0:(cl_io.c:214:cl_io_rw_init()) header@000000004699652a[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760797.923542:0:11778:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28733440, 28737536) 0 0 00000080:00200000:3.0:1789760797.923547:0:11778:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760797.923575:0:11778:0:(file.c:2168:ll_file_io_generic()) Range lock [7015, 7015] 00020000:00200000:3.0:1789760797.923585:0:11778:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28733440 chunk: [28733440, 28737536) 28737536, 4096 00020000:00200000:3.0:1789760797.923590:0:11778:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760797.923611:0:11778:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7015, 7015] 00000008:00010000:3.0:1789760797.923632:0:11778:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760797.923639:0:11778:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8eb650f27d30, osc lock ffff8eb64610da20, flags 0x0 00000080:00200000:3.0:1789760797.923652:0:11778:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28733440, 28737536) append 0 append lockpos 0 00000080:00200000:3.0:1789760797.923667:0:11778:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8eb64999ba88), size=4096 (max 536870912), offset=28733440=0x1b67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760797.923735:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760797.923738:0:11778:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28737536 00000020:00010000:3.0:1789760797.923759:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_release():245 00000020:00010000:3.0:1789760797.923763:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.923767:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8eb650f27d30 (ffff8eb650f97cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760797.923771:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb644dac218 (ffff8eb6450b4908/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760797.923773:0:11778:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8eb650f27d30 (ffff8eb6450b4908/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760797.923782:0:11778:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760797.923790:0:11778:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.923813:0:33:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8eb649d27900 00000008:00100000:3.0:1789760797.923936:0:33:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8eb64514aa00 - object 0x240000400:5 offset 28733440<>28737536 00000100:00100000:3.0:1789760797.923991:0:2370:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.924014:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493884416:192.168.203.160@tcp:4:dd.0 00010000:00010000:3.0:1789760797.933730:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760798.254781:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493884672:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760798.255184:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493884544:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760798.255253:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493884800:192.168.203.160@tcp:400:kworker.0 00010000:00010000:0.0:1789760798.956624:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760799.981449:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789760800.301205:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1789760800.301229:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760800.301303:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1789760801.004797:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760802.029012:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760803.052587:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760803.372592:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493885056:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760803.372603:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493884928:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760803.372680:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493885312:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760803.372682:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493885184:192.168.203.160@tcp:400:kworker.0 00010000:00010000:3.0:1789760804.076583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760805.100573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760806.124560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760807.148569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760808.172533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760808.492566:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493885440:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760808.492598:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493885568:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760808.492652:0:2370:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493885824:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760808.492656:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493885696:192.168.203.160@tcp:400:kworker.0 00010000:00010000:3.0:1789760809.196542:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760810.220563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760811.244542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760812.268605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760813.292552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760813.612597:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493885952:192.168.203.160@tcp:400:kworker.0 00000100:00000400:3.0:1789760813.612623:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760797/real 1789760797] req@ffff8eb64514aa00 x1876699493884416/t0(0) o4->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789760813 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1789760813.612693:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493886208:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760813.612719:0:2369:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1789760813.612726:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493886080:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760813.612763:0:2369:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493886336:192.168.203.160@tcp:400:kworker.0 00000100:00080000:3.0:1789760813.620165:0:2370:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760813.620172:0:2370:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8eb6513ab000: Connection to lustre-OST0000 (at 192.168.203.160@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760813.623562:0:2370:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760813.623623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760813.623628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760813.623630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760813.623635:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 400 00000100:00080000:1.0:1789760813.623645:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789760813.623655:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760813.623674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493886464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760813.623715:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886464:192.168.203.160@tcp:8:kworker.0 00010000:00010000:0.0:1789760814.316606:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1789760814.636501:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760798/real 1789760798] req@ffff8eb645149180 x1876699493884672/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1789760814.636501:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760798/real 1789760798] req@ffff8eb64514b800 x1876699493884544/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760814.636557:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb645149180 x1876699493884672/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760814.636567:0:2369:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1789760814.636570:0:2369:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8eb6513ab000: Connection to lustre-MDT0000 (at 192.168.203.160@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:3.0:1789760814.643919:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb64514b800 x1876699493884544/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760814.643931:0:2370:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1789760814.643935:0:2370:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.160@tcp: Connection to MGS (at 192.168.203.160@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1789760814.649589:0:2369:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1789760814.649627:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493884672:192.168.203.160@tcp:400:kworker.0 00000100:00080000:3.0:1789760814.654721:0:2370:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.160@tcp for MGC192.168.203.160@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1789760814.654724:0:2370:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1789760814.654729:0:2370:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8eb64514b800 x1876699493884544/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760814.654738:0:2370:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8eb650f82a00 x1876699493884928/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760819 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760814.654750:0:2370:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8eb64514b480 x1876699493885440/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760824 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760814.654756:0:2370:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8eb645148700 x1876699493885952/t0(0) o400->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760829 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760814.654764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:3.0:1789760814.654768:0:2370:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1789760814.654772:0:2370:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 00000100:00080000:1.0:1789760814.654774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 10000000:01000000:3.0:1789760814.654775:0:2370:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1789760814.654779:0:2370:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760814.654780:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760814.654798:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 258 00000100:00100000:2.0:1789760814.654804:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493885952:192.168.203.160@tcp:400:kworker.0 00000100:00080000:1.0:1789760814.654812:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00000400:3.0:1789760814.654812:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760798/real 1789760798] req@ffff8eb645148000 x1876699493884800/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760814.654826:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493885440:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760814.654842:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96f72786-7ebd-4df8-8048-c06aee14e87f:2369:1876699493884928:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760814.654843:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb645148000 x1876699493884800/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760814 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760814.654847:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493886592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760814.654867:0:2370:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760814.654873:0:2370:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8eb648433000: Connection to lustre-MDT0000 (at 192.168.203.160@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760814.654877:0:2370:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1789760814.654893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789760814.654896:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96f72786-7ebd-4df8-8048-c06aee14e87f:2370:1876699493884544:192.168.203.160@tcp:400:kworker.0 00000100:00080000:1.0:1789760814.654902:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760814.654907:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760814.654909:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493884800:192.168.203.160@tcp:400:kworker.0 00000100:00080000:1.0:1789760814.654916:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 258 00000100:00080000:1.0:1789760814.654924:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760814.654928:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493886592:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789760814.654932:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760814.654946:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493886720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760814.654956:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760814.654959:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760814.654961:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760814.654963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760814.654965:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760814.654967:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 326 00000100:00080000:1.0:1789760814.654971:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789760814.654975:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760814.654982:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493886848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760814.654994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760814.654996:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760814.655000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760814.655002:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760814.655015:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760814.655043:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493886848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760815.340525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760816.364565:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760817.388589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760818.412519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760818.668507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.668518:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.668522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.668526:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.668529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.668532:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.668535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.668539:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760819.436546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760819.692469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760819.692477:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760819.692481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760819.692483:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760819.692485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760819.692488:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760819.692490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760819.692494:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1789760819.756497:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760803/real 1789760803] req@ffff8eb650f80700 x1876699493885184/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1789760819.756498:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760803/real 1789760803] req@ffff8eb650f81880 x1876699493885056/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760819.756551:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb650f80700 x1876699493885184/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760819.756562:0:2369:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8eb6450cf800 already not connected (conn 4, was 3): CONNECTING 00000100:00100000:3.0:1789760819.756577:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493885184:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760819.774958:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb650f81880 x1876699493885056/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760819.774971:0:2370:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6462fb000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1789760819.774980:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760803/real 1789760803] req@ffff8eb650f81c00 x1876699493885312/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760819.774995:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb650f81c00 x1876699493885312/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760819 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760819.775005:0:2370:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6431a3000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1789760819.775019:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493885056:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760819.775026:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493885312:192.168.203.160@tcp:400:kworker.0 00010000:00010000:0.0:1789760820.460544:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760821.484572:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760822.508561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760823.532578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760823.788479:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8eb65016d880 x1876699493886464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760823.788508:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493886464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760823.788525:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760823.788546:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760823.788550:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886464:192.168.203.160@tcp:8:kworker.0 00000100:00080000:3.0:1789760823.788591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.788597:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760823.788601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.788604:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760823.788606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.788609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760823.788611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760823.788615:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 823 00000100:00080000:3.0:1789760823.788625:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760823.788630:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760823.788639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760823.788665:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493886976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760823.788688:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.788691:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760823.788730:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886976:192.168.203.160@tcp:8:kworker.0 00010000:00010000:3.0:1789760824.556574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760824.812524:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8eb65016e680 x1876699493886720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.812564:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493886720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.812573:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8eb65016f800 x1876699493886848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.812580:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493886848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.812586:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760814/real 1789760814] req@ffff8eb65016c380 x1876699493886592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.812592:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493886592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760824.812606:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760824.812625:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760824.812629:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886720:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789760824.812633:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760824.812637:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760824.812639:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493886848:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789760824.812642:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760824.812645:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760824.812648:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493886592:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760824.812676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760824.812683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760824.812687:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760824.812694:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 824 00000100:00080000:3.0:1789760824.812704:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760824.812723:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760824.812769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493887104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760824.812792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760824.812795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760824.812797:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760824.812800:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 824 00000100:00080000:3.0:1789760824.812804:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760824.812807:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760824.812816:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760824.812824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493887104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760824.812829:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493887232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760824.812838:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760824.812841:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760824.812844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760824.812846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760824.812848:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760824.812850:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 824 00000100:00080000:3.0:1789760824.812853:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760824.812855:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760824.812858:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760824.812865:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493887360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760824.812911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760824.812940:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493887360:192.168.203.160@tcp:38:kworker.0 00000100:00000400:2.0:1789760824.876486:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760808/real 1789760808] req@ffff8eb64514ad80 x1876699493885568/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1789760824.876542:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760808/real 1789760808] req@ffff8eb645148380 x1876699493885696/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760824.876567:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb645148380 x1876699493885696/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760824.876628:0:2369:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8eb6450cf800 already not connected (conn 5, was 3): CONNECTING 00000100:00100000:3.0:1789760824.876642:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493885696:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760824.885530:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb64514ad80 x1876699493885568/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760824.885538:0:2370:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6462fb000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1789760824.885544:0:2370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760808/real 1789760808] req@ffff8eb64514bb80 x1876699493885824/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760824.885552:0:2370:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb64514bb80 x1876699493885824/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760824.885558:0:2370:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6431a3000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1789760824.885566:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2370:1876699493885568:192.168.203.160@tcp:400:kworker.0 00000100:00100000:2.0:1789760824.885570:0:2370:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2370:1876699493885824:192.168.203.160@tcp:400:kworker.0 00010000:00010000:3.0:1789760825.580551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760826.604570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760827.628736:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760828.652640:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760828.844559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760828.844569:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760828.844573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760828.844577:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760828.844579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760828.844582:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760828.844585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760828.844589:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760829.676717:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760829.868519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760829.868527:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760829.868531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760829.868550:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760829.868553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760829.868556:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760829.868558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760829.868560:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1789760829.996495:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8eb645148e00 x1876699493886080/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760830.010576:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb645148e00 x1876699493886080/t0(0) o400->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760830.010607:0:2369:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6462fb000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:3.0:1789760830.010618:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8eb64514a300 x1876699493886336/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760830.010636:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb64514a300 x1876699493886336/t0(0) o400->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760830.010647:0:2369:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8eb6431a3000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:3.0:1789760830.010655:0:2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8eb645149500 x1876699493886208/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760830.010707:0:2369:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8eb645149500 x1876699493886208/t0(0) o400->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760830.010713:0:2369:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8eb6450cf800 already not connected (conn 5, was 3): CONNECTING 00000100:00100000:3.0:1789760830.010725:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493886080:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760830.010730:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2369:1876699493886336:192.168.203.160@tcp:400:kworker.0 00000100:00100000:3.0:1789760830.010733:0:2369:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2369:1876699493886208:192.168.203.160@tcp:400:kworker.0 00010000:00010000:3.0:1789760830.700621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760831.724740:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760832.748672:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760833.772658:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760834.796612:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760835.116539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760835.116547:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760835.116550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760835.116553:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760835.116556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760835.116559:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760835.116562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760835.116565:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760835.820589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760836.844743:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760837.868687:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760838.060540:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760823/real 1789760823] req@ffff8eb645148000 x1876699493886976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760838.060586:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493886976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760838 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760838.060636:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760838.060663:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760838.060668:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493886976:192.168.203.160@tcp:8:kworker.0 00000100:00080000:0.0:1789760838.060714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.060722:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760838.060725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.060728:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760838.060730:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.060733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760838.060736:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760838.060742:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 833 00000100:00080000:0.0:1789760838.060752:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760838.060760:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760838.060770:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760838.060802:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493887488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760838.060829:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.060833:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760838.060875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887488:192.168.203.160@tcp:8:kworker.0 00010000:00010000:0.0:1789760838.892653:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760839.084585:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760824/real 1789760824] req@ffff8eb645149880 x1876699493887232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760839.084670:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493887232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760839.084683:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760824/real 1789760824] req@ffff8eb64514b100 x1876699493887360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760839.084694:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493887360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760839.084720:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760824/real 1789760824] req@ffff8eb64514b800 x1876699493887104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760839.084730:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493887104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760839 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760839.084748:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760839.084775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760839.084780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887232:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789760839.084785:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760839.084789:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760839.084793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493887360:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789760839.084797:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760839.084803:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760839.084807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493887104:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760839.084827:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760839.084844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760839.084847:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760839.084857:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 834 00000100:00080000:0.0:1789760839.084869:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760839.084877:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760839.084942:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493887616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760839.085003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760839.085011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760839.085015:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760839.085025:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 834 00000100:00080000:0.0:1789760839.085032:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760839.085041:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760839.085056:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760839.085059:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493887616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760839.085093:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493887744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760839.085121:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760839.085129:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760839.085136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760839.085143:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760839.085147:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760839.085155:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 834 00000100:00080000:0.0:1789760839.085165:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760839.085174:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760839.085186:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760839.085209:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493887872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760839.085236:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887744:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760839.085291:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493887872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760839.916619:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760840.940669:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760841.964733:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760842.988594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760843.116584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.116592:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760843.116597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.116600:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760843.116603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.116606:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760843.116615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760843.116618:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760844.012613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760844.140532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760844.140544:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760844.140549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760844.140552:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760844.140557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760844.140560:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760844.140580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760844.140584:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760845.036609:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760846.060622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760847.084688:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760848.108508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760849.132683:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760849.452649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760849.452662:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760849.452667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760849.452675:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760849.452680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760849.452684:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760849.452689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760849.452694:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760850.156648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760851.180571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760852.204680:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760853.228628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760854.252611:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.1F:1789760854.316522:0:2358:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.203.160@tcp (192.168.203.160:988) the network or that node may be down. 00000100:00080000:2.0:1789760854.572508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760854.572525:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760854.572529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760854.572533:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760854.572547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760854.572551:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760854.572555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760854.572559:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760855.276654:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760856.300559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760857.324640:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760858.348540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760858.476566:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760838/real 1789760838] req@ffff8eb64304dc00 x1876699493887488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760858.476629:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493887488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760858 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760858.476669:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760858.476705:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760858.476721:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887488:192.168.203.160@tcp:8:kworker.0 00000100:00080000:2.0:1789760858.476775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760858.476793:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760858.476797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760858.476800:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760858.476802:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760858.476804:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760858.476818:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760858.476823:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 847 00000100:00080000:2.0:1789760858.476832:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760858.476838:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760858.476846:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760858.476884:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493888000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760858.476904:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760858.476907:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789760858.476936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888000:192.168.203.160@tcp:8:kworker.0 00010000:00010000:3.0:1789760859.372580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760859.500652:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760839/real 1789760839] req@ffff8eb64304ce00 x1876699493887872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760859.500686:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493887872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760859.500707:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760839/real 1789760839] req@ffff8eb64304f800 x1876699493887744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760859.500730:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493887744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760859.500741:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760839/real 1789760839] req@ffff8eb64304d500 x1876699493887616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760859.500754:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493887616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760859 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760859.500773:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760859.500797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760859.500802:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493887872:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789760859.500808:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760859.500814:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760859.500818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493887744:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789760859.500822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760859.500827:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760859.500829:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493887616:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760859.500832:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760859.500839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760859.500842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760859.500851:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 848 00000100:00080000:2.0:1789760859.500863:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760859.500871:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760859.500902:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493888128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760859.500925:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760859.500929:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760859.500940:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760859.500943:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 848 00000100:00080000:2.0:1789760859.500946:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760859.500950:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760859.500959:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760859.500968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493888128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760859.500973:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493888256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760859.500983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760859.500986:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760859.500988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760859.500991:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760859.500992:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760859.500995:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 848 00000100:00080000:2.0:1789760859.500997:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760859.501009:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760859.501013:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760859.501020:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493888384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760859.501058:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760859.501096:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493888384:192.168.203.160@tcp:38:kworker.0 00000400:00000100:0.0:1789760860.204677:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.160:988 00000400:00000100:0.0:1789760860.204731:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760860.204749:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760860.204790:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760860.204797:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760860.204803:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760860.204877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760859/real 1789760860] req@ffff8eb650f82a00 x1876699493888256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.204911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493888256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760860.204942:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760860.204948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760860.204953:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760860.204964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760859/real 1789760860] req@ffff8eb650f82d80 x1876699493888384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.204987:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493888384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760860.204995:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760860.205000:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760860.205004:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493888384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760860.205011:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760859/real 1789760860] req@ffff8eb650f81880 x1876699493888128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.205024:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493888128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760860.205031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760860.205035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760860.205039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493888128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760860.205046:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760858/real 1789760860] req@ffff8eb650f81c00 x1876699493888000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760860.205059:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493888000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760860.205067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760860.205071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760860.205074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888000:192.168.203.160@tcp:8:kworker.0 00010000:00010000:0.0:1789760860.396532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760861.420549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760862.444662:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760863.468607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760863.532510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760863.532523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760863.532528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760863.532538:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 869 00000100:00080000:2.0:1789760863.532548:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760863.532555:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760863.532599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493888512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760863.532625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760863.532628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760863.532630:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760863.532634:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 869 00000100:00080000:2.0:1789760863.532637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760863.532640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760863.532649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760863.532661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493888640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760863.532671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760863.532674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760863.532675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760863.532678:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 868 00000100:00080000:2.0:1789760863.532682:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760863.532687:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493888512:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760863.532688:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760863.532698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760863.532711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493888768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760863.532726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760863.532731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760863.532733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760863.532738:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 869 00000100:00080000:2.0:1789760863.532754:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760863.532761:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760863.532768:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760863.532787:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493888896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760863.532796:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760863.532830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760863.532867:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493888896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760864.492704:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760865.517072:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760866.348782:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.160:988 00000400:00000100:0.0:1789760866.348840:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760866.348858:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760866.348896:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760866.348906:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760866.348920:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760866.348953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760863/real 1789760866] req@ffff8eb650f81880 x1876699493888768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.348987:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493888768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.349004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.349012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760866.349019:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760866.349030:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760863/real 1789760866] req@ffff8eb650f82d80 x1876699493888896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.349057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493888896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.349069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.349075:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760866.349080:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493888896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760866.349089:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760863/real 1789760866] req@ffff8eb650f82680 x1876699493888512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.349106:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493888512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.349117:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.349122:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760866.349126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493888512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760866.349136:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760863/real 1789760866] req@ffff8eb650f81c00 x1876699493888640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760866.349152:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493888640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760866.349163:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760866.349168:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760866.349173:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493888640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760866.540579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760867.565153:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760868.588884:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760868.908518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760868.908530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760868.908536:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760868.908550:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 873 00000100:00080000:2.0:1789760868.908563:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760868.908572:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760868.908622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493889024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760868.908649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760868.908653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760868.908655:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760868.908659:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 873 00000100:00080000:2.0:1789760868.908662:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760868.908665:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760868.908677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760868.908687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493889152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760868.908696:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760868.908699:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760868.908701:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760868.908706:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 873 00000100:00080000:2.0:1789760868.908710:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760868.908716:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760868.908723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760868.908753:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493889280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760868.908770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493889024:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760868.908775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760868.908792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760868.908794:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760868.908800:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 873 00000100:00080000:2.0:1789760868.908806:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760868.908812:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760868.908820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760868.908849:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493889408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760868.908922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760868.908958:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760868.908981:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493889408:192.168.203.160@tcp:38:kworker.0 00000400:00000100:1.0:1789760869.421093:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.160:988 00000400:00000100:1.0:1789760869.421181:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760869.421207:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789760869.421278:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789760869.421328:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760869.421331:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760868/real 1789760869] req@ffff8eb650f82a00 x1876699493889024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1789760869.421343:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760869.421370:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493889024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760869.421428:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760869.421435:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760869.421447:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493889024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760869.421463:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760868/real 1789760869] req@ffff8eb650f80700 x1876699493889152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760869.421483:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493889152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760869.421493:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760869.421498:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760869.421503:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760869.421528:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760868/real 1789760869] req@ffff8eb650f80e00 x1876699493889280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760869.421543:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493889280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760869.421552:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760869.421556:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760869.421559:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760869.421565:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760868/real 1789760869] req@ffff8eb650f83480 x1876699493889408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760869.421576:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493889408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760869.421583:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760869.421587:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760869.421591:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493889408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760869.613005:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760870.636669:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760871.660670:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760872.684620:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760873.708644:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760874.028490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760874.028498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760874.028501:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760874.028509:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 878 00000100:00080000:3.0:1789760874.028518:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760874.028523:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760874.028555:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493889536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760874.028582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760874.028584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760874.028586:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760874.028589:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 878 00000100:00080000:3.0:1789760874.028593:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760874.028596:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760874.028604:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760874.028613:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493889664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760874.028621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760874.028626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789760874.028628:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493889536:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760874.028629:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760874.028637:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 878 00000100:00080000:3.0:1789760874.028645:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760874.028649:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760874.028652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760874.028663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493889792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760874.028672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760874.028674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760874.028675:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760874.028678:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 878 00000100:00080000:3.0:1789760874.028681:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760874.028685:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760874.028688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760874.028697:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493889920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760874.028759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760874.028783:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760874.028803:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493889920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760874.732821:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760875.564613:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.160:988 00000400:00000100:0.0:1789760875.564653:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760875.564666:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760875.564705:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760875.564720:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789760875.564731:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760874/real 1789760875] req@ffff8eb64514a300 x1876699493889664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1789760875.564736:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789760875.564770:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493889664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760875.564783:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760875.564788:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760875.564798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760875.564807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760874/real 1789760875] req@ffff8eb645148e00 x1876699493889792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760875.564823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493889792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760875.564836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760875.564840:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760875.564844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493889792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760875.564851:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760874/real 1789760875] req@ffff8eb645148380 x1876699493889920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760875.564865:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493889920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760875.564874:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760875.564878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760875.564881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493889920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760875.564889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760874/real 1789760875] req@ffff8eb645149500 x1876699493889536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760875.564902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493889536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760875.564910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760875.564914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760875.564918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493889536:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789760875.756554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760876.780708:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760877.804542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760878.828769:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760879.148539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760879.148548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760879.148551:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760879.148559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 883 00000100:00080000:0.0:1789760879.148568:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760879.148575:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760879.148616:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493890048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760879.148635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760879.148638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760879.148639:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760879.148642:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 883 00000100:00080000:0.0:1789760879.148645:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760879.148647:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760879.148654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760879.148662:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493890176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760879.148675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760879.148677:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760879.148679:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760879.148682:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 883 00000100:00080000:0.0:1789760879.148684:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789760879.148687:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493890048:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760879.148689:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760879.148697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760879.148704:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493890304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760879.148711:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760879.148714:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760879.148715:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760879.148717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 883 00000100:00080000:0.0:1789760879.148720:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760879.148723:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760879.148726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760879.148735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493890432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789760879.148762:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760879.148807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760879/real 1789760879] req@ffff8eb64304d880 x1876699493890048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760879.148829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493890048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760879.148841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760879.148846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760879.148850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493890048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760879.148858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760879.148877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760879.148887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760879/real 1789760879] req@ffff8eb64304f480 x1876699493890176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760879.148896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493890176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760879.148901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760879.148903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760879.148906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760879.148925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760879.148936:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760879.148943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760879/real 1789760879] req@ffff8eb64304e680 x1876699493890304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760879.148950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493890304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760879.148955:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760879.148957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760879.148959:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760879.148962:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493890432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760879.148972:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760879.148977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760879/real 1789760879] req@ffff8eb64304f100 x1876699493890432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760879.148984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493890432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760879.148989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760879.148991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760879.148992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493890432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760879.852575:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760880.876722:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760881.900583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760882.924604:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760883.948711:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760884.268502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760884.268513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760884.268518:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760884.268529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 888 00000100:00080000:3.0:1789760884.268542:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760884.268549:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760884.268592:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493890560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760884.268617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760884.268620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760884.268623:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760884.268626:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 888 00000100:00080000:3.0:1789760884.268629:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760884.268632:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760884.268643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760884.268652:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493890688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760884.268661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760884.268665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789760884.268667:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493890560:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760884.268667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760884.268685:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 888 00000100:00080000:3.0:1789760884.268689:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760884.268692:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760884.268697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760884.268716:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493890816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760884.268728:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760884.268732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760884.268734:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760884.268739:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 888 00000100:00080000:3.0:1789760884.268745:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760884.268751:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760884.268757:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760884.268774:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493890944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760884.268793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760884.268823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760884.268844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493890944:192.168.203.160@tcp:38:kworker.0 00000400:00000100:0.0:1789760884.780698:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.160:988 00000400:00000100:0.0:1789760884.780752:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760884.780769:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760884.780808:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760884.780819:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760884.780829:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789760884.780855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760884/real 1789760884] req@ffff8eb64514a680 x1876699493890688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760884.780884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493890688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760884.780897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760884.780903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760884.780908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760884.780917:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760884/real 1789760884] req@ffff8eb645148a80 x1876699493890816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760884.780932:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493890816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760884.780941:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760884.780945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760884.780949:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493890816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760884.780956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760884/real 1789760884] req@ffff8eb645149500 x1876699493890944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760884.780970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493890944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760884.780987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760884.780991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760884.780995:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493890944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760884.781003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760884/real 1789760884] req@ffff8eb645149c00 x1876699493890560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760884.781015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493890560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760884.781039:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760884.781043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760884.781047:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493890560:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789760884.972574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760885.996645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760887.020624:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760888.044592:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760889.068607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760889.388485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760889.388497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760889.388502:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760889.388513:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 893 00000100:00080000:3.0:1789760889.388532:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760889.388539:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760889.388570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493891072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760889.388594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760889.388597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760889.388598:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760889.388601:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 893 00000100:00080000:3.0:1789760889.388605:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760889.388609:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760889.388618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760889.388633:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493891200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760889.388644:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493891072:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760889.388648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760889.388654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760889.388656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760889.388661:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 893 00000100:00080000:3.0:1789760889.388665:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760889.388669:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760889.388675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760889.388689:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493891328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789760889.388701:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789760889.388702:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760889.388707:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760889.388710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760889.388716:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 893 00000100:00080000:3.0:1789760889.388721:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760889.388725:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760889.388729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760889.388745:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760889/real 1789760889] req@ffff8eb645148380 x1876699493891072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760889.388746:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493891456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760889.388790:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493891072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760889.388800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760889.388804:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760889.388807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493891072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760889.388826:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760889.388845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760889.388853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760889/real 1789760889] req@ffff8eb645148e00 x1876699493891200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760889.388861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493891200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760889.388866:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760889.388869:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760889.388871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760889.388874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760889.388885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760889.388891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760889/real 1789760889] req@ffff8eb64514a300 x1876699493891328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760889.388898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493891328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760889.388903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760889.388905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760889.388907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760889.388910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493891456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760889.388919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760889.388925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760889/real 1789760889] req@ffff8eb64514b800 x1876699493891456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760889.388937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493891456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760889.388942:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760889.388943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760889.388945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493891456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760890.092896:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760891.116845:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760892.140511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760893.164997:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760894.188900:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760894.508519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760894.508530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760894.508535:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760894.508545:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 899 00000100:00080000:0.0:1789760894.508557:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760894.508565:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760894.508607:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493891584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760894.508640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760894.508644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760894.508646:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760894.508651:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 899 00000100:00080000:0.0:1789760894.508656:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760894.508662:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760894.508675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760894.508687:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493891712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760894.508698:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493891584:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760894.508707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760894.508713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760894.508716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760894.508724:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 899 00000100:00080000:0.0:1789760894.508729:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760894.508735:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760894.508744:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760894.508756:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493891840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760894.508765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789760894.508771:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789760894.508778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760894.508781:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760894.508789:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 899 00000100:00080000:0.0:1789760894.508797:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760894.508800:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760894.508804:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760894.508820:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304df80 x1876699493891968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760894.508834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760894/real 1789760894] req@ffff8eb64304ca80 x1876699493891584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760894.508851:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493891584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760894.508863:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760894.508868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760894.508872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493891584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760894.508892:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760894.508908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760894.508916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760894/real 1789760894] req@ffff8eb64304ea00 x1876699493891712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760894.508929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493891712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760894.508944:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760894.508948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760894.508952:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760894.508959:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760894.508988:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760894.508997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760894/real 1789760894] req@ffff8eb64304c700 x1876699493891840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760894.509005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493891840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760894.509011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760894.509013:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760894.509015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493891840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760894.509019:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493891968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760894.509032:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760894.509037:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760894/real 1789760894] req@ffff8eb64304df80 x1876699493891968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760894.509044:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304df80 x1876699493891968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760894.509049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760894.509051:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760894.509054:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493891968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760895.212872:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760896.236601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760897.260689:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760898.284588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760899.308582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760899.628456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760899.628464:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760899.628467:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760899.628476:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 904 00000100:00080000:0.0:1789760899.628484:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760899.628491:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760899.628523:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304df80 x1876699493892096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760899.628544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760899.628547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760899.628548:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760899.628552:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 904 00000100:00080000:0.0:1789760899.628554:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760899.628557:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760899.628564:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760899.628572:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493892224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760899.628579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760899.628581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760899.628582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760899.628584:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 904 00000100:00080000:0.0:1789760899.628585:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760899.628587:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760899.628590:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760899.628596:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493892352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760899.628603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760899.628605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760899.628606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760899.628609:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 904 00000100:00080000:0.0:1789760899.628611:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760899.628614:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760899.628618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760899.628620:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493892096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760899.628627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493892480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760899.628679:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760899.628711:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760899/real 1789760899] req@ffff8eb64304df80 x1876699493892096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760899.628734:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304df80 x1876699493892096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760899.628742:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760899.628745:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760899.628748:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493892096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760899.628752:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760899.628767:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760899.628774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760899/real 1789760899] req@ffff8eb64304c700 x1876699493892224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760899.628782:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493892224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760899.628787:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760899.628789:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760899.628791:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760899.628795:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892352:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760899.628805:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760899.628811:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760899/real 1789760899] req@ffff8eb64304ea00 x1876699493892352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760899.628829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493892352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760899.628837:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760899.628841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760899.628844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760899.628870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493892480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760899.628890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760899.628899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760899/real 1789760899] req@ffff8eb64304ca80 x1876699493892480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760899.628907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493892480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760899.628913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760899.628915:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760899.628917:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493892480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760900.333000:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760901.356552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760902.380678:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760903.404809:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760904.428879:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760904.748514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760904.748529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760904.748534:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760904.748548:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 909 00000100:00080000:2.0:1789760904.748561:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760904.748570:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760904.748615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493892608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760904.748639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760904.748642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760904.748644:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760904.748647:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 909 00000100:00080000:2.0:1789760904.748650:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760904.748652:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760904.748662:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760904.748672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493892736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760904.748679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760904.748682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760904.748684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760904.748686:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 909 00000100:00080000:2.0:1789760904.748689:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760904.748692:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760904.748698:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493892608:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789760904.748701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760904.748718:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493892864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760904.748731:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760904.748736:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760904.748738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760904.748743:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 909 00000100:00080000:2.0:1789760904.748747:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760904.748751:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760904.748755:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760904.748765:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493892992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760904.748824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760904.748872:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760904.748902:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493892992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760905.452604:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760906.284973:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789760906.285033:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760906.285051:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760906.285090:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760906.285151:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760904/real 1789760906] req@ffff8eb650f81180 x1876699493892608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760906.285181:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493892608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760906.285190:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760906.285194:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760906.285197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493892608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760906.285203:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760904/real 1789760906] req@ffff8eb650f83800 x1876699493892736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760906.285211:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493892736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760906.285217:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760906.285219:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760906.285221:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789760906.285316:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789760906.285343:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760906.285384:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760904/real 1789760906] req@ffff8eb650f83b80 x1876699493892864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760906.285434:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493892864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760906.285450:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760906.285454:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760906.285458:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493892864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760906.285466:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760904/real 1789760906] req@ffff8eb650f81f80 x1876699493892992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760906.285490:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493892992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760906.285502:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760906.285507:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760906.285511:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493892992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760906.476788:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760907.500776:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760908.524755:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760909.548617:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760909.868453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760909.868461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760909.868465:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760909.868474:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 914 00000100:00080000:2.0:1789760909.868483:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760909.868489:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760909.868519:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493893120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760909.868539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760909.868542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760909.868544:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760909.868547:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 914 00000100:00080000:2.0:1789760909.868550:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760909.868552:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760909.868560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760909.868568:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493893248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760909.868603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760909.868608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760909.868611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760909.868617:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 914 00000100:00080000:2.0:1789760909.868622:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760909.868627:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760909.868635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760909.868639:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493893120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760909.868653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493893376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760909.868664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760909.868667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760909.868670:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760909.868673:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 914 00000100:00080000:2.0:1789760909.868691:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760909.868694:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760909.868699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760909.868701:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760909.868715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493893504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760909.868746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760909/real 1789760909] req@ffff8eb650f80a80 x1876699493893120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760909.868768:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493893120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760909.868780:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760909.868785:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760909.868790:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493893120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760909.868799:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760909.868824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760909.868835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760909/real 1789760909] req@ffff8eb650f80000 x1876699493893248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760909.868845:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493893248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760909.868851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760909.868854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760909.868856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760909.868875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760909.868887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760909.868894:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760909/real 1789760909] req@ffff8eb650f80380 x1876699493893376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760909.868902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493893376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760909.868908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760909.868910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760909.868912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760909.868916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493893504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760909.868944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760909.868950:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760909/real 1789760909] req@ffff8eb650f81500 x1876699493893504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760909.868957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493893504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760909.868962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760909.868964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760909.868966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493893504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760910.572827:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760911.596532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760912.620627:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760913.644668:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760914.668727:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760914.988607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760914.988619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760914.988625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760914.988653:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 919 00000100:00080000:0.0:1789760914.988667:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760914.988677:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760914.988726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493893632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760914.988768:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760914.988773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760914.988776:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760914.988781:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 919 00000100:00080000:0.0:1789760914.988786:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760914.988792:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760914.988805:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760914.988822:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493893760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760914.988829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493893632:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760914.988865:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760914.988870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760914.988873:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760914.988879:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 919 00000100:00080000:0.0:1789760914.988883:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760914.988888:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760914.988893:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760914.988906:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493893888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760914.988918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760914.988922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760914.988927:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760914.988933:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 919 00000100:00080000:0.0:1789760914.988938:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760914.988949:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:3.0:1789760914.988949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789760914.988966:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760914.988985:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493894016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760914.989027:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760914/real 1789760914] req@ffff8eb64304f100 x1876699493893632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760914.989073:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493893632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760914.989085:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760914.989089:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760914.989092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493893632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760914.989128:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760914.989159:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760914.989174:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760914/real 1789760914] req@ffff8eb64304e680 x1876699493893760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760914.989185:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493893760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760914.989194:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760914.989198:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760914.989202:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760914.989210:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893888:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760914.989233:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760914.989242:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760914/real 1789760914] req@ffff8eb64304f480 x1876699493893888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760914.989251:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493893888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760914.989257:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760914.989259:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760914.989262:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493893888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760914.989267:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493894016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760914.989279:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760914.989286:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760914/real 1789760914] req@ffff8eb64304d880 x1876699493894016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760914.989294:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493894016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760914.989300:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760914.989302:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760914.989305:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493894016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760915.692729:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760916.716656:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760917.740905:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760918.764518:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760919.788623:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760920.108548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760920.108556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760920.108561:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760920.108570:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 924 00000100:00080000:0.0:1789760920.108580:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760920.108586:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760920.108630:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493894144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760920.108649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760920.108652:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760920.108654:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760920.108657:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 924 00000100:00080000:0.0:1789760920.108659:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760920.108683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760920.108691:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760920.108701:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493894144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760920.108704:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493894272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760920.108715:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760920.108718:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760920.108719:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760920.108722:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 924 00000100:00080000:0.0:1789760920.108725:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760920.108727:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760920.108731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760920.108740:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493894400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760920.108748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760920.108757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760920.108760:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789760920.108761:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789760920.108766:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 924 00000100:00080000:0.0:1789760920.108770:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760920.108774:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760920.108778:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760920.108788:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493894528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760920.108800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760920/real 1789760920] req@ffff8eb64304d500 x1876699493894144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760920.108815:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493894144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760920.108824:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760920.108828:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760920.108831:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493894144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760920.108869:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760920.108885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760920.108895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760920/real 1789760920] req@ffff8eb64304f800 x1876699493894272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760920.108903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493894272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760920.108921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760920.108923:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760920.108925:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760920.108929:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760920.108940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760920.108946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760920/real 1789760920] req@ffff8eb64304ce00 x1876699493894400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760920.108953:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493894400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760920.108958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760920.108960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760920.108962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760920.108966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493894528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760920.108975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760920.108981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760920/real 1789760920] req@ffff8eb64304dc00 x1876699493894528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760920.108989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493894528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760920.108993:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760920.109010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760920.109015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493894528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760920.813026:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760921.836961:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760922.861076:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760923.884628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760924.908647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760925.228499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760925.228507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760925.228510:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760925.228518:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 929 00000100:00080000:2.0:1789760925.228527:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760925.228532:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760925.228566:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493894656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760925.228589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760925.228592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760925.228593:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760925.228596:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 929 00000100:00080000:2.0:1789760925.228598:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760925.228601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760925.228608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760925.228616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493894784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760925.228623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760925.228626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760925.228627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760925.228630:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 929 00000100:00080000:2.0:1789760925.228632:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760925.228636:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493894656:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760925.228637:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760925.228645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760925.228653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493894912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760925.228672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760925.228674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760925.228676:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760925.228679:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 929 00000100:00080000:2.0:1789760925.228681:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760925.228683:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760925.228686:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760925.228693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493895040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760925.228713:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760925.228746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760925/real 1789760925] req@ffff8eb650f81500 x1876699493894656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760925.228759:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493894656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760925.228767:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760925.228769:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760925.228772:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493894656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760925.228777:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760925.228792:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760925.228799:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760925/real 1789760925] req@ffff8eb650f80380 x1876699493894784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760925.228807:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493894784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760925.228812:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760925.228815:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760925.228817:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760925.228835:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894912:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760925.228846:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760925.228853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760925/real 1789760925] req@ffff8eb650f80000 x1876699493894912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760925.228860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493894912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760925.228865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760925.228867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760925.228869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493894912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760925.228873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493895040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760925.228882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760925.228888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760925/real 1789760925] req@ffff8eb650f80a80 x1876699493895040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760925.228895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493895040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760925.228900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760925.228902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760925.228904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493895040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760925.932533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760926.956787:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760927.980512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760929.004519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760930.028531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760930.348515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760930.348525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760930.348530:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760930.348542:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 934 00000100:00080000:2.0:1789760930.348554:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760930.348562:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760930.348594:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493895168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760930.348619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760930.348622:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760930.348623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760930.348626:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 934 00000100:00080000:2.0:1789760930.348630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760930.348633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760930.348640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760930.348650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493895296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760930.348659:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760930.348661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760930.348665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760930.348666:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493895168:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760930.348670:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 934 00000100:00080000:2.0:1789760930.348675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760930.348677:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760930.348681:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760930.348692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493895424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760930.348699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760930.348702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760930.348703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760930.348706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 934 00000100:00080000:2.0:1789760930.348717:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760930.348720:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760930.348723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760930.348728:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760930.348736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493895552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760930.348766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760930/real 1789760930] req@ffff8eb650f81f80 x1876699493895168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760930.348779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493895168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760930.348786:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760930.348789:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760930.348792:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493895168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760930.348810:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760930.348825:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760930.348832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760930/real 1789760930] req@ffff8eb650f83b80 x1876699493895296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760930.348840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493895296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760930.348846:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760930.348848:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760930.348850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760930.348854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760930.348866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760930.348872:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760930/real 1789760930] req@ffff8eb650f83800 x1876699493895424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760930.348880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493895424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760930.348885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760930.348886:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760930.348888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760930.348891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493895552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760930.348900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760930.348906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760930/real 1789760930] req@ffff8eb650f81180 x1876699493895552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760930.348913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493895552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760930.348933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760930.348936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760930.348940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493895552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760931.052559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760932.076525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760933.100550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760934.124497:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760935.148491:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760935.468539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760935.468549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760935.468554:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760935.468565:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 939 00000100:00080000:2.0:1789760935.468577:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760935.468586:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760935.468624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493895680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760935.468654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760935.468658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760935.468661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760935.468665:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 939 00000100:00080000:2.0:1789760935.468669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760935.468674:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760935.468684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760935.468699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493895680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760935.468702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493895808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760935.468718:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760935.468722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760935.468724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760935.468728:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 939 00000100:00080000:2.0:1789760935.468732:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760935.468737:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760935.468743:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760935.468755:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493895936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760935.468767:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789760935.468768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760935.468774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760935.468777:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760935.468783:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 939 00000100:00080000:2.0:1789760935.468791:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760935.468796:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760935.468802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760935.468808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760935/real 1789760935] req@ffff8eb650f83480 x1876699493895680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760935.468817:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493896064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760935.468827:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493895680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760935.468838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760935.468842:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760935.468846:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493895680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760935.468866:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760935.468896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760935.468906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760935/real 1789760935] req@ffff8eb650f80e00 x1876699493895808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760935.468915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493895808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760935.468921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760935.468923:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760935.468925:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760935.468935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760935.468950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760935.468957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760935/real 1789760935] req@ffff8eb650f80700 x1876699493895936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760935.468970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493895936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760935.468975:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760935.468977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760935.468979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493895936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760935.468983:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493896064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760935.468993:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760935.468999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760935/real 1789760935] req@ffff8eb650f82a00 x1876699493896064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760935.469007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493896064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760935.469012:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760935.469013:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760935.469015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493896064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760936.172499:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760937.196523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760938.220526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760939.244520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760940.268523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760940.588463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760940.588471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760940.588475:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760940.588484:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 945 00000100:00080000:2.0:1789760940.588493:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760940.588499:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760940.588531:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493896192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760940.588548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760940.588552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760940.588554:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760940.588558:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 945 00000100:00080000:2.0:1789760940.588561:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760940.588564:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760940.588572:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493896192:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789760940.588579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760940.588592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493896320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760940.588601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760940.588603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760940.588605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760940.588609:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 945 00000100:00080000:2.0:1789760940.588619:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760940.588621:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760940.588625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760940.588634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493896448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760940.588645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760940.588648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760940.588649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760940.588652:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 945 00000100:00080000:2.0:1789760940.588655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760940.588661:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760940.588661:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896320:192.168.203.160@tcp:38:kworker.0 00000008:00100000:2.0:1789760940.588669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760940.588681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493896576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760940.588694:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760940.588728:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493896576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760941.292510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760942.316515:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760943.340503:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760943.660639:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789760943.660684:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760943.660698:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789760943.660730:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789760943.660748:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789760943.660757:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789760943.660790:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760940/real 1789760943] req@ffff8eb650f81880 x1876699493896576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.660817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493896576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.660842:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.660848:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760943.660852:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493896576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760943.660860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760940/real 1789760943] req@ffff8eb650f82680 x1876699493896320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.660874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493896320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.660882:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.660887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760943.660891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760943.660898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760940/real 1789760943] req@ffff8eb650f82d80 x1876699493896448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.660924:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493896448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.660935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.660938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760943.660941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760943.660948:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760940/real 1789760943] req@ffff8eb650f81c00 x1876699493896192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.660961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493896192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.660969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.660972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760943.660975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493896192:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789760944.364532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760945.388511:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760945.708503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760945.708511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760945.708514:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760945.708523:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 950 00000100:00080000:2.0:1789760945.708532:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760945.708538:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760945.708575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493896704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760945.708598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760945.708603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760945.708605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760945.708608:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 950 00000100:00080000:2.0:1789760945.708610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760945.708613:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760945.708620:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760945.708629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493896832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760945.708636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760945.708639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760945.708640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760945.708643:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 950 00000100:00080000:2.0:1789760945.708645:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760945.708648:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760945.708652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760945.708658:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493896704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760945.708661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493896960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760945.708674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760945.708676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760945.708678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760945.708680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 950 00000100:00080000:2.0:1789760945.708682:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760945.708684:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760945.708688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760945.708695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493897088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760945.708733:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760945.708774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760945/real 1789760945] req@ffff8eb650f81c00 x1876699493896704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760945.708793:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493896704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760945.708819:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760945.708825:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760945.708830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493896704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760945.708838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760945.708863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760945.708877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760945/real 1789760945] req@ffff8eb650f82d80 x1876699493896832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760945.708891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493896832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760945.708900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760945.708904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760945.708907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760945.708931:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760945.708958:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760945.708969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760945/real 1789760945] req@ffff8eb650f82680 x1876699493896960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760945.708982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493896960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760945.708991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760945.708994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760945.708997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493896960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760945.709003:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493897088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760945.709022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760945.709032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760945/real 1789760945] req@ffff8eb650f81880 x1876699493897088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760945.709044:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493897088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760945.709051:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760945.709055:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760945.709058:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493897088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760946.412533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760947.436515:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760948.460520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760949.484506:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760950.508519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760950.828496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760950.828506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760950.828511:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760950.828522:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 955 00000100:00080000:3.0:1789760950.828533:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760950.828541:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760950.828578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493897216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760950.828610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760950.828614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760950.828616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760950.828622:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 955 00000100:00080000:3.0:1789760950.828627:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760950.828633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760950.828642:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493897216:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789760950.828644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760950.828663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493897344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760950.828675:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760950.828679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760950.828681:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760950.828686:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 955 00000100:00080000:3.0:1789760950.828691:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760950.828695:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:2.0:1789760950.828699:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789760950.828704:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760950.828720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493897472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760950.828732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760950.828735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789760950.828739:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760950/real 1789760950] req@ffff8eb64514b100 x1876699493897216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760950.828740:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760950.828749:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 955 00000100:00080000:3.0:1789760950.828757:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760950.828763:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760950.828782:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493897216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789760950.828786:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789760950.828794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760950.828799:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760950.828803:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493897216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760950.828807:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493897600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760950.828830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760950.828852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760950.828861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760950/real 1789760950] req@ffff8eb645149880 x1876699493897344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760950.828870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493897344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760950.828876:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760950.828878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760950.828880:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760950.828884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897472:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760950.828896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760950.828902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760950/real 1789760950] req@ffff8eb645148000 x1876699493897472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760950.828909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493897472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760950.828915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760950.828916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760950.828918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760950.828922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493897600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760950.828932:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760950.828938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760950/real 1789760950] req@ffff8eb64514bb80 x1876699493897600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760950.828946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493897600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760950.828951:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760950.828952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760950.828954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493897600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760951.532563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760952.556556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760953.580521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760954.604532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760955.628591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760955.948516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760955.948524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760955.948528:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760955.948536:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 960 00000100:00080000:0.0:1789760955.948545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760955.948550:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760955.948582:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493897728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760955.948606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760955.948610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760955.948612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760955.948616:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 960 00000100:00080000:0.0:1789760955.948620:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760955.948624:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760955.948634:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760955.948647:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493897856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760955.948649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493897728:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760955.948659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760955.948662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760955.948663:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760955.948667:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 960 00000100:00080000:0.0:1789760955.948670:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760955.948673:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760955.948677:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760955.948686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493897984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760955.948695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760955.948698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760955.948699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760955.948702:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 960 00000800:00000100:2.0:1789760955.948706:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789760955.948711:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760955.948716:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760955.948720:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760955.948751:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493898112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760955.948760:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760955/real 1789760955] req@ffff8eb64304c380 x1876699493897728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760955.948777:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493897728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760955.948797:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760955.948802:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760955.948806:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493897728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760955.948828:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760955.948847:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760955.948857:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760955/real 1789760955] req@ffff8eb64304c000 x1876699493897856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760955.948865:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493897856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760955.948875:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760955.948877:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760955.948879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760955.948882:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760955.948893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760955.948898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760955/real 1789760955] req@ffff8eb64304d180 x1876699493897984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760955.948905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493897984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760955.948911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760955.948913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760955.948914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493897984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760955.948918:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493898112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760955.948928:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760955.948934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760955/real 1789760955] req@ffff8eb64304ed80 x1876699493898112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760955.948941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493898112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760955.948945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760955.948947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760955.948950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493898112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760956.652621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760957.676613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760958.700888:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760959.724600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760960.748524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760961.068546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760961.068556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760961.068559:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760961.068567:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 965 00000100:00080000:0.0:1789760961.068577:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760961.068582:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760961.068622:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493898240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760961.068649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760961.068652:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760961.068653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760961.068656:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 965 00000100:00080000:0.0:1789760961.068670:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760961.068673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760961.068680:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760961.068700:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493898240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760961.068706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493898368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760961.068716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760961.068719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760961.068720:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760961.068723:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 965 00000100:00080000:0.0:1789760961.068725:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760961.068727:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760961.068731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760961.068757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493898496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760961.068764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760961.068766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760961.068768:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760961.068770:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 965 00000100:00080000:0.0:1789760961.068773:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760961.068775:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760961.068779:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789760961.068781:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760961.068789:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493898624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760961.068824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760961/real 1789760961] req@ffff8eb64304e300 x1876699493898240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760961.068844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493898240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760961.068860:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760961.068865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760961.068871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493898240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760961.068896:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760961.068919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760961.068931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760961/real 1789760961] req@ffff8eb64304ed80 x1876699493898368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760961.068943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493898368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760961.068950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760961.068953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760961.068956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760961.068962:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760961.068982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760961.068995:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760961/real 1789760961] req@ffff8eb64304d180 x1876699493898496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760961.069005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493898496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760961.069010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760961.069012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760961.069014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760961.069018:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493898624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760961.069033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760961.069039:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760961/real 1789760961] req@ffff8eb64304c000 x1876699493898624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760961.069047:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493898624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760961.069052:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760961.069054:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760961.069056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493898624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760961.772584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760962.796569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760963.820534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760964.844505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760965.868513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760966.188446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760966.188454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760966.188457:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760966.188465:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 970 00000100:00080000:0.0:1789760966.188474:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760966.188497:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760966.188538:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493898752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760966.188582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760966.188586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760966.188587:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760966.188591:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 970 00000100:00080000:0.0:1789760966.188594:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760966.188597:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760966.188606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760966.188617:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493898880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760966.188642:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760966.188645:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760966.188662:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760966.188664:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493898752:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789760966.188667:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 970 00000100:00080000:0.0:1789760966.188675:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760966.188677:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760966.188681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760966.188692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493899008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760966.188701:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760966.188703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760966.188705:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760966.188707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 970 00000100:00080000:0.0:1789760966.188710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760966.188712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760966.188715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789760966.188726:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760966.188727:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493899136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760966.188762:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760966/real 1789760966] req@ffff8eb64304c380 x1876699493898752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760966.188777:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493898752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760966.188785:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760966.188788:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760966.188791:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493898752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760966.188809:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760966.188824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760966.188832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760966/real 1789760966] req@ffff8eb64304dc00 x1876699493898880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760966.188840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493898880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760966.188845:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760966.188848:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760966.188850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493898880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760966.188854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899008:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760966.188864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760966.188869:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760966/real 1789760966] req@ffff8eb64304ce00 x1876699493899008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760966.188877:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493899008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760966.188882:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760966.188884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760966.188886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760966.188890:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493899136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760966.188900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760966.188906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760966/real 1789760966] req@ffff8eb64304f800 x1876699493899136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760966.188913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493899136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760966.188918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760966.188920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760966.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493899136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760966.892550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760967.916627:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760968.940586:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760969.964603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760970.988548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760971.308558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760971.308572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760971.308586:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760971.308601:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 975 00000100:00080000:0.0:1789760971.308630:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760971.308656:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760971.308722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493899264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760971.308756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760971.308760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760971.308762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760971.308768:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 975 00000100:00080000:0.0:1789760971.308773:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760971.308779:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760971.308794:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760971.308803:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493899264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760971.308816:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493899392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760971.308833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760971.308838:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760971.308864:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760971.308870:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 975 00000100:00080000:0.0:1789760971.308877:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789760971.308883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789760971.308887:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760971.308898:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760971.308914:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493899520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760971.308927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760971.308932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760971.308940:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760971.308940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760971/real 1789760971] req@ffff8eb64304d500 x1876699493899264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760971.308955:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 975 00000100:00080000:0.0:1789760971.308972:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789760971.308979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493899264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760971.308983:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760971.309000:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789760971.309003:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760971.309011:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760971.309015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493899264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760971.309026:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493899648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760971.309042:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760971.309062:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760971.309073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760971/real 1789760971] req@ffff8eb64304d880 x1876699493899392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760971.309093:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493899392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760971.309100:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760971.309103:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760971.309106:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760971.309111:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760971.309129:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760971.309150:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760971/real 1789760971] req@ffff8eb64304f480 x1876699493899520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760971.309170:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493899520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760971.309179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760971.309184:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760971.309187:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760971.309210:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493899648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760971.309221:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760971.309227:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760971/real 1789760971] req@ffff8eb64304e680 x1876699493899648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760971.309234:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493899648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760971.309243:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760971.309245:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760971.309247:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493899648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760972.012538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760973.036546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760974.060558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760975.084620:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760976.108542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760976.428548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760976.428556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760976.428559:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760976.428567:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 980 00000100:00080000:2.0:1789760976.428587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760976.428593:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760976.428623:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493899776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760976.428646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760976.428649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760976.428650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760976.428653:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 980 00000100:00080000:2.0:1789760976.428656:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760976.428658:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760976.428670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760976.428679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493899904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760976.428687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760976.428690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760976.428692:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760976.428697:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493899776:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789760976.428698:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 980 00000100:00080000:2.0:1789760976.428703:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760976.428707:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760976.428711:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760976.428719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493900032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760976.428726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760976.428729:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760976.428730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760976.428733:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 980 00000100:00080000:2.0:1789760976.428750:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760976.428754:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789760976.428760:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789760976.428761:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760976.428776:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493900160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760976.428818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760976/real 1789760976] req@ffff8eb650f82a00 x1876699493899776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760976.428842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493899776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760976.428853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760976.428858:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760976.428862:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493899776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760976.428885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760976.428911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760976.428926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760976/real 1789760976] req@ffff8eb650f80700 x1876699493899904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760976.428940:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493899904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760976.428949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760976.428952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760976.428956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493899904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789760976.428963:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789760976.428983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760976.428993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760976/real 1789760976] req@ffff8eb650f80e00 x1876699493900032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760976.429000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493900032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760976.429015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760976.429018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760976.429022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789760976.429028:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493900160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789760976.429049:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789760976.429061:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760976/real 1789760976] req@ffff8eb650f83480 x1876699493900160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760976.429070:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493900160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760976.429075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760976.429077:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789760976.429079:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493900160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760977.132596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760978.156536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760979.180579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760980.204539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760981.228671:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760981.548492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760981.548503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760981.548508:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760981.548519:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 986 00000100:00080000:3.0:1789760981.548531:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760981.548551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760981.548588:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493900288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760981.548619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760981.548623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760981.548625:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760981.548632:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 986 00000100:00080000:3.0:1789760981.548637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760981.548642:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789760981.548650:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493900288:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789760981.548652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760981.548682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493900416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760981.548694:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760981.548698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760981.548700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760981.548704:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 986 00000100:00080000:3.0:1789760981.548709:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760981.548713:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:2.0:1789760981.548718:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789760981.548721:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760981.548738:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493900544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760981.548750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789760981.548753:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760981/real 1789760981] req@ffff8eb64514ad80 x1876699493900288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760981.548755:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760981.548760:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760981.548770:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 986 00000100:00100000:2.0:1789760981.548774:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493900288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760981.548776:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760981.548787:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:2.0:1789760981.548788:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789760981.548794:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789760981.548795:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760981.548800:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493900288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789760981.548815:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493900672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760981.548822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760981.548845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760981.548856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760981/real 1789760981] req@ffff8eb64514b480 x1876699493900416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760981.548869:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493900416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760981.548877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760981.548881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760981.548885:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760981.548892:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900544:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760981.548914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760981.548924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760981/real 1789760981] req@ffff8eb645148700 x1876699493900544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760981.548933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493900544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760981.548938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760981.548940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760981.548942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760981.548962:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493900672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760981.548975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760981.548981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760981/real 1789760981] req@ffff8eb645149180 x1876699493900672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760981.548989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493900672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760981.548994:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760981.548996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760981.548998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493900672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760982.252555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760983.276638:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760984.300657:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760985.324661:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760986.348528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760986.668528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760986.668539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760986.668554:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760986.668566:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 991 00000100:00080000:3.0:1789760986.668580:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760986.668589:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760986.668648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493900800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760986.668687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760986.668691:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760986.668693:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760986.668697:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 991 00000100:00080000:3.0:1789760986.668701:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760986.668705:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760986.668715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760986.668731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493900928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760986.668759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789760986.668765:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493900800:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760986.668767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760986.668774:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760986.668784:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 991 00000100:00080000:3.0:1789760986.668790:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760986.668795:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760986.668815:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760986.668839:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493901056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789760986.668850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789760986.668858:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760986.668864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760986.668866:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760986.668872:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 991 00000100:00080000:3.0:1789760986.668877:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760986.668881:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760986.668901:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760986.668923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760986/real 1789760986] req@ffff8eb645149180 x1876699493900800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760986.668932:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493901184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760986.668943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493900800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760986.668966:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760986.668969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760986.668972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493900800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760986.668995:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760986.669018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760986.669037:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760986/real 1789760986] req@ffff8eb645148700 x1876699493900928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760986.669051:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493900928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760986.669058:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760986.669060:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760986.669062:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493900928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760986.669066:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760986.669082:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760986.669105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760986/real 1789760986] req@ffff8eb64514b480 x1876699493901056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760986.669116:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493901056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760986.669124:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760986.669127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760986.669130:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760986.669136:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493901184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760986.669154:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760986.669164:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760986/real 1789760986] req@ffff8eb64514ad80 x1876699493901184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760986.669185:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493901184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760986.669198:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760986.669201:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760986.669204:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493901184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760987.372593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760988.396569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760989.420630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760990.444602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760991.468606:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760991.788499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760991.788508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760991.788512:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760991.788520:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 996 00000100:00080000:3.0:1789760991.788530:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760991.788536:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789760991.788566:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493901312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760991.788592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760991.788595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760991.788596:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760991.788599:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 996 00000100:00080000:3.0:1789760991.788602:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760991.788604:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760991.788612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760991.788622:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493901440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760991.788631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789760991.788634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493901312:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789760991.788635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760991.788638:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760991.788646:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 996 00000100:00080000:3.0:1789760991.788648:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760991.788651:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760991.788655:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760991.788665:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493901568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760991.788682:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760991.788685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760991.788686:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760991.788689:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 996 00000100:00080000:3.0:1789760991.788692:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760991.788694:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789760991.788697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760991.788704:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493901696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789760991.788711:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760991.788749:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760991/real 1789760991] req@ffff8eb64514bb80 x1876699493901312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760991.788762:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493901312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760991.788770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760991.788772:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789760991.788775:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493901312:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760991.788794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789760991.788810:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760991.788818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760991/real 1789760991] req@ffff8eb645148000 x1876699493901440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760991.788826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493901440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760991.788832:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760991.788834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789760991.788844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789760991.788848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901568:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789760991.788861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760991.788867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760991/real 1789760991] req@ffff8eb645149880 x1876699493901568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760991.788875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493901568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760991.788880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760991.788882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789760991.788884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789760991.788888:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493901696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789760991.788897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789760991.788903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760991/real 1789760991] req@ffff8eb64514b100 x1876699493901696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760991.788909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493901696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760991.788914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760991.788916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789760991.788918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493901696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789760992.492559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760993.516561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760994.540650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760995.564579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760996.588544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760996.908510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760996.908518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760996.908521:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760996.908529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1001 00000100:00080000:0.0:1789760996.908538:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760996.908544:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789760996.908574:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493901824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760996.908598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760996.908602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760996.908604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760996.908608:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1001 00000100:00080000:0.0:1789760996.908611:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760996.908615:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760996.908625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760996.908639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493901952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760996.908651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760996.908655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760996.908657:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760996.908661:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1001 00000100:00080000:0.0:1789760996.908664:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760996.908668:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760996.908673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760996.908675:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493901824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789760996.908684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493902080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760996.908694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760996.908697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760996.908698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760996.908701:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1001 00000100:00080000:0.0:1789760996.908703:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760996.908706:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789760996.908709:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760996.908717:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493902208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789760996.908732:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760996.908766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760996/real 1789760996] req@ffff8eb64304f100 x1876699493901824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760996.908779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493901824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760996.908787:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760996.908790:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760996.908793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493901824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789760996.908797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760996.908810:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760996.908818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760996/real 1789760996] req@ffff8eb64304ca80 x1876699493901952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760996.908825:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493901952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760996.908830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760996.908832:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760996.908834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493901952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789760996.908852:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789760996.908863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760996.908869:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760996/real 1789760996] req@ffff8eb64304ea00 x1876699493902080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760996.908876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493902080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760996.908881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760996.908883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760996.908885:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789760996.908889:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493902208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789760996.908898:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789760996.908903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760996/real 1789760996] req@ffff8eb64304c700 x1876699493902208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760996.908911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493902208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760996.908916:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760996.908917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789760996.908919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493902208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:2.0:1789760997.612504:0:11846:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1789760997.612573:0:5607:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1789760997.612581:0:5607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb64165fa00/0xa279945d11e7246e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x3eb3dcc0b2dfb600 expref: -99 pid: 11778 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789760997.612596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1789760997.612607:0:5607:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1789760997.612627:0:5607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb640030e00/0xa279945d11e72467 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x3eb3dcc0b2dfb5e4 expref: -99 pid: 11778 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760997.612637:0:5607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb650164000/0xa279945d11e72452 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x3eb3dcc0b2dfb582 expref: -99 pid: 11771 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789760997.612671:0:5607:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb64165fa00/0xa279945d11e7246e lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x3eb3dcc0b2dfb600 expref: -99 pid: 11778 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760997.612681:0:5607:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb640030e00/0xa279945d11e72467 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x3eb3dcc0b2dfb5e4 expref: -99 pid: 11778 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760997.612685:0:5607:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb650164000/0xa279945d11e72452 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x3eb3dcc0b2dfb582 expref: -99 pid: 11771 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789760997.612689:0:5607:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1789760997.612720:0:5607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb64165fa00/0xa279945d11e7246e lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x3eb3dcc0b2dfb600 expref: -99 pid: 11778 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760997.612739:0:5607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb640030e00/0xa279945d11e72467 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x3eb3dcc0b2dfb5e4 expref: -99 pid: 11778 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789760997.612750:0:5607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8eb6513ab000 lock: ffff8eb650164000/0xa279945d11e72452 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x3eb3dcc0b2dfb582 expref: -99 pid: 11771 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00080000:1.0:1789760997.612758:0:2368:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8eb64304df80 x1876699493902336/t0(0) o103->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1789760997.612762:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760997.612766:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760997.612769:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760998.636606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760999.660594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761000.684563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761001.708547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761002.028570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761002.028580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761002.028585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761002.028595:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1006 00000100:00080000:3.0:1789761002.028607:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761002.028615:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761002.028656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493902464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761002.028687:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761002.028692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761002.028694:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761002.028699:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1006 00000100:00080000:3.0:1789761002.028703:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761002.028708:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761002.028718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761002.028733:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493902592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761002.028739:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493902464:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761002.028754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761002.028759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761002.028761:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761002.028767:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1006 00000100:00080000:3.0:1789761002.028770:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761002.028775:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761002.028781:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761002.028795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493902720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761002.028809:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761002.028813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761002.028816:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761002.028823:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761002.028825:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1006 00000100:00080000:3.0:1789761002.028832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761002.028838:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761002.028844:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761002.028858:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493902848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761002.028875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761002/real 1789761002] req@ffff8eb64514b800 x1876699493902464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761002.028900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493902464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761002.028914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761002.028919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761002.028925:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493902464:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761002.028953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761002.028995:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761002.029010:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761002/real 1789761002] req@ffff8eb64514a300 x1876699493902592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761002.029027:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493902592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761002.029037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761002.029042:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761002.029046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902592:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761002.029054:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902720:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761002.029081:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761002.029095:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761002/real 1789761002] req@ffff8eb645148e00 x1876699493902720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761002.029111:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493902720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761002.029122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761002.029127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761002.029131:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493902720:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761002.029140:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493902848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761002.029165:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761002.029180:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761002/real 1789761002] req@ffff8eb645148380 x1876699493902848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761002.029195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493902848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761002.029205:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761002.029209:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761002.029214:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493902848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761002.732614:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761003.756536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761004.780570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761005.804535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761006.828510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761007.148486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761007.148494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761007.148498:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761007.148506:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1011 00000100:00080000:2.0:1789761007.148515:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761007.148520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761007.148552:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493902976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761007.148575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761007.148578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761007.148579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761007.148582:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1011 00000100:00080000:2.0:1789761007.148584:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761007.148587:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761007.148594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761007.148603:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493903104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761007.148610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761007.148613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761007.148614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761007.148616:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1011 00000100:00080000:2.0:1789761007.148618:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761007.148621:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761007.148624:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761007.148626:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493902976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761007.148632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493903232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761007.148646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761007.148648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761007.148650:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761007.148653:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1011 00000100:00080000:2.0:1789761007.148655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761007.148657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761007.148660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761007.148668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493903360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761007.148728:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761007.148769:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761007.148791:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493903360:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761007.852538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761008.876503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761009.900552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761010.220651:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761010.220691:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761010.220708:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761010.220750:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761010.220761:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761010.220771:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761010.220800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761007/real 1789761010] req@ffff8eb650f83b80 x1876699493903232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761010.220826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493903232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761010.220840:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761010.220845:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761010.220849:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761010.220858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761007/real 1789761010] req@ffff8eb650f81f80 x1876699493903360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761010.220871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493903360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761010.220880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761010.220884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761010.220888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493903360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761010.220894:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761007/real 1789761010] req@ffff8eb650f81180 x1876699493902976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761010.220907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493902976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761010.220915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761010.220919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761010.220922:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493902976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761010.220929:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761007/real 1789761010] req@ffff8eb650f83800 x1876699493903104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761010.220940:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493903104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761010.220949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761010.220952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761010.220955:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761010.924640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761011.948531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761012.268478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761012.268487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761012.268490:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761012.268498:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1016 00000100:00080000:2.0:1789761012.268507:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761012.268513:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761012.268541:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493903488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761012.268566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761012.268569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761012.268570:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761012.268573:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1016 00000100:00080000:2.0:1789761012.268575:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761012.268577:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761012.268585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761012.268593:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493903616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761012.268601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761012.268604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761012.268605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761012.268607:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1016 00000100:00080000:2.0:1789761012.268610:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761012.268612:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761012.268615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761012.268622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493903744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761012.268629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761012.268632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761012.268633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761012.268636:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1016 00000100:00080000:2.0:1789761012.268638:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761012.268641:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761012.268645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761012.268653:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493903488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761012.268664:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493903872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761012.268735:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761012.268790:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761012/real 1789761012] req@ffff8eb650f80a80 x1876699493903488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761012.268813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493903488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761012.268827:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761012.268832:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761012.268836:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493903488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761012.268845:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761012.268871:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761012.268886:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761012/real 1789761012] req@ffff8eb650f80000 x1876699493903616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761012.268900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493903616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761012.268908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761012.268912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761012.268915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903616:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761012.268922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903744:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761012.268946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761012.268959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761012/real 1789761012] req@ffff8eb650f80380 x1876699493903744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761012.268971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493903744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761012.268979:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761012.268982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761012.268985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493903744:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761012.269020:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493903872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761012.269043:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761012.269056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761012/real 1789761012] req@ffff8eb650f81500 x1876699493903872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761012.269070:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493903872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761012.269078:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761012.269082:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761012.269085:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493903872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761012.972558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761013.996530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761015.020544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761016.044570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761017.068545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761017.388526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761017.388536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761017.388540:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761017.388561:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1021 00000100:00080000:3.0:1789761017.388605:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761017.388629:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761017.388726:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493904000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761017.388770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761017.388792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761017.388795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761017.388811:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493904000:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761017.388835:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1021 00000100:00080000:3.0:1789761017.388841:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761017.388867:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789761017.388879:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789761017.388897:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761017.388923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761017/real 1789761017] req@ffff8eb645149c00 x1876699493904000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761017.388925:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493904128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761017.388954:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493904000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761017.388957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761017.388966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761017.388972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.388976:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761017.388979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493904000:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761017.388988:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761017.388995:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1021 00000100:00100000:1.0:1789761017.388999:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904128:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789761017.389019:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761017.389030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761017.389045:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761017/real 1789761017] req@ffff8eb645149500 x1876699493904128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761017.389047:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761017.389058:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761017.389063:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493904128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.389071:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.389073:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761017.389075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761017.389086:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493904256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761017.389102:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761017.389107:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761017.389109:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761017.389114:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904256:192.168.203.160@tcp:8:kworker.0 00000100:00080000:3.0:1789761017.389115:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1021 00000100:00080000:3.0:1789761017.389123:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761017.389128:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761017.389133:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761017.389146:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493904384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761017.389148:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761017.389161:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761017/real 1789761017] req@ffff8eb645148a80 x1876699493904256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.389171:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493904256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.389179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.389181:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761017.389184:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904256:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761017.389193:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493904384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761017.389205:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761017.389212:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761017/real 1789761017] req@ffff8eb64514a680 x1876699493904384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761017.389219:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493904384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761017.389225:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761017.389227:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761017.389229:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493904384:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761018.092525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761019.116553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761020.140605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761021.164524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761022.188581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761022.508487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761022.508495:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761022.508498:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761022.508507:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1027 00000100:00080000:2.0:1789761022.508516:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761022.508522:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761022.508563:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493904512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761022.508583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761022.508586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761022.508587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761022.508590:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1027 00000100:00080000:2.0:1789761022.508598:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761022.508601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761022.508609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761022.508618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493904640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761022.508626:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761022.508629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761022.508630:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761022.508634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493904512:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761022.508636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1027 00000100:00080000:2.0:1789761022.508642:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761022.508645:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761022.508648:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761022.508657:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493904768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761022.508668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761022.508671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761022.508672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761022.508675:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1027 00000100:00080000:2.0:1789761022.508677:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761022.508679:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761022.508682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761022.508690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493904896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761022.508695:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761022.508741:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761022/real 1789761022] req@ffff8eb650f81500 x1876699493904512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761022.508761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493904512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761022.508769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761022.508773:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761022.508777:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493904512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761022.508795:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761022.508814:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761022.508823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761022/real 1789761022] req@ffff8eb650f80380 x1876699493904640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761022.508832:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493904640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761022.508838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761022.508840:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761022.508842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761022.508846:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904768:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761022.508858:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761022.508864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761022/real 1789761022] req@ffff8eb650f80000 x1876699493904768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761022.508872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493904768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761022.508877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761022.508878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761022.508880:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493904768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761022.508884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493904896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761022.508893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761022.508899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761022/real 1789761022] req@ffff8eb650f80a80 x1876699493904896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761022.508909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493904896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761022.508919:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761022.508921:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761022.508923:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493904896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761023.212548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761024.236559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761025.260521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761026.284574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761027.308517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761027.628501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761027.628510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761027.628515:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761027.628524:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1032 00000100:00080000:3.0:1789761027.628536:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761027.628543:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761027.628580:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493905024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761027.628607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761027.628611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761027.628614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761027.628618:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1032 00000100:00080000:3.0:1789761027.628621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761027.628625:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761027.628634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761027.628647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493905152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761027.628661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761027.628665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761027.628667:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761027.628671:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1032 00000100:00080000:3.0:1789761027.628674:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761027.628680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761027.628688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761027.628690:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493905024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761027.628702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493905280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761027.628715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761027.628719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761027.628722:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761027.628726:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1032 00000100:00080000:3.0:1789761027.628730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761027.628733:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761027.628738:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761027.628751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493905408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761027.628784:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761027.628852:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761027/real 1789761027] req@ffff8eb64514a680 x1876699493905024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761027.628875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493905024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761027.628888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761027.628894:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761027.628898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493905024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761027.628908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761027.628935:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761027.628949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761027/real 1789761027] req@ffff8eb645148a80 x1876699493905152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761027.628962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493905152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761027.628971:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761027.628975:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761027.628978:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761027.629009:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905280:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761027.629036:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761027.629050:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761027/real 1789761027] req@ffff8eb645149500 x1876699493905280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761027.629064:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493905280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761027.629074:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761027.629078:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761027.629082:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761027.629089:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493905408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761027.629110:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761027.629122:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761027/real 1789761027] req@ffff8eb645149c00 x1876699493905408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761027.629131:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493905408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761027.629137:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761027.629139:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761027.629142:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493905408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761028.332536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761029.356501:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761030.380515:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761031.404555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761032.428514:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761032.748545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761032.748552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761032.748556:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761032.748565:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1037 00000100:00080000:2.0:1789761032.748574:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761032.748580:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761032.748608:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493905536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761032.748628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761032.748631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761032.748632:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761032.748635:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1037 00000100:00080000:2.0:1789761032.748638:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761032.748640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761032.748647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761032.748656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493905664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761032.748663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761032.748666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761032.748667:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761032.748671:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1037 00000100:00080000:2.0:1789761032.748675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761032.748679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761032.748684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761032.748692:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493905536:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761032.748700:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493905792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761032.748722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761032.748726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761032.748727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761032.748730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1037 00000100:00080000:2.0:1789761032.748734:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761032.748737:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761032.748741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761032.748753:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493905920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761032.748761:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761032.748807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761032/real 1789761032] req@ffff8eb650f83800 x1876699493905536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761032.748828:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493905536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761032.748839:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761032.748844:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761032.748848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493905536:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761032.748856:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761032.748882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761032.748908:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761032/real 1789761032] req@ffff8eb650f81180 x1876699493905664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761032.748921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493905664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761032.748930:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761032.748933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761032.748937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761032.748961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905792:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761032.749002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761032.749015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761032/real 1789761032] req@ffff8eb650f81f80 x1876699493905792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761032.749030:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493905792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761032.749048:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761032.749052:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761032.749057:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493905792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761032.749064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493905920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761032.749091:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761032.749106:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761032/real 1789761032] req@ffff8eb650f83b80 x1876699493905920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761032.749122:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493905920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761032.749131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761032.749135:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761032.749139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493905920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761033.452543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761034.476533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761035.500548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761036.524550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761037.548591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761037.868519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761037.868534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761037.868540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761037.868561:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1042 00000100:00080000:0.0:1789761037.868576:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761037.868586:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761037.868636:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493906048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761037.868662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761037.868665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761037.868667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761037.868670:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1042 00000100:00080000:0.0:1789761037.868673:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761037.868676:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761037.868686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761037.868698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493906176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761037.868707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761037.868710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493906048:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761037.868711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761037.868726:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761037.868732:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1042 00000100:00080000:0.0:1789761037.868734:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761037.868737:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761037.868741:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761037.868751:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493906304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761037.868758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761037.868761:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761037.868763:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761037.868767:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1042 00000100:00080000:0.0:1789761037.868799:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761037.868802:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761037.868806:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761037.868814:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493906432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761037.868823:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761037.868894:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761037/real 1789761037] req@ffff8eb64304c700 x1876699493906048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761037.868912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493906048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761037.868949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761037.868953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761037.868956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493906048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761037.868981:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761037.869000:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761037.869020:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761037/real 1789761037] req@ffff8eb64304ea00 x1876699493906176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761037.869029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493906176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761037.869035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761037.869038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761037.869040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761037.869044:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761037.869057:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761037.869063:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761037/real 1789761037] req@ffff8eb64304ca80 x1876699493906304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761037.869082:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493906304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761037.869087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761037.869089:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761037.869091:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761037.869096:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493906432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761037.869107:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761037.869129:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761037/real 1789761037] req@ffff8eb64304f100 x1876699493906432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761037.869167:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493906432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761037.869178:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761037.869180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761037.869182:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493906432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761038.572560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761039.596535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761040.620532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761041.644544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761042.668513:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761042.988501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761042.988509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761042.988513:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761042.988521:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1047 00000100:00080000:2.0:1789761042.988530:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761042.988537:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761042.988582:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493906560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761042.988612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761042.988615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761042.988617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761042.988620:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1047 00000100:00080000:2.0:1789761042.988623:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761042.988627:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761042.988635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761042.988647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493906688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761042.988656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761042.988660:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493906560:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761042.988660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761042.988661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761042.988667:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1047 00000100:00080000:2.0:1789761042.988672:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761042.988675:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761042.988678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761042.988686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493906816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761042.988698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761042.988700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761042.988702:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761042.988704:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1047 00000100:00080000:2.0:1789761042.988707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761042.988709:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761042.988712:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761042.988719:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493906944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761042.988736:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761042.988776:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761042/real 1789761042] req@ffff8eb650f83480 x1876699493906560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761042.988791:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493906560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761042.988800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761042.988805:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761042.988811:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493906560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761042.988834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761042.988859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761042.988869:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761042/real 1789761042] req@ffff8eb650f80e00 x1876699493906688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761042.988878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493906688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761042.988884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761042.988886:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761042.988889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761042.988893:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906816:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761042.988906:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761042.988912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761042/real 1789761042] req@ffff8eb650f80700 x1876699493906816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761042.988920:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493906816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761042.988925:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761042.988927:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761042.988929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493906816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761042.988933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493906944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761042.988942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761042.988948:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761042/real 1789761042] req@ffff8eb650f82a00 x1876699493906944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761042.988955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493906944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761042.988964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761042.988967:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761042.988969:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493906944:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761043.692590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761044.716503:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761045.740586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761046.764505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761047.788534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761048.108496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761048.108505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761048.108509:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761048.108517:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1052 00000100:00080000:3.0:1789761048.108526:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761048.108532:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761048.108565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493907072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761048.108587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761048.108589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761048.108591:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761048.108594:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1052 00000100:00080000:3.0:1789761048.108596:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761048.108598:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761048.108606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761048.108614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493907200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761048.108622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761048.108624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761048.108626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761048.108628:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1052 00000100:00080000:3.0:1789761048.108630:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761048.108632:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761048.108636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761048.108640:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493907072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761048.108645:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493907328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761048.108655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761048.108658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761048.108659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761048.108662:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1052 00000100:00080000:3.0:1789761048.108664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761048.108666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761048.108668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761048.108674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493907456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761048.108715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761048.108746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761048/real 1789761048] req@ffff8eb645148380 x1876699493907072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761048.108761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493907072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761048.108769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761048.108772:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761048.108774:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493907072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761048.108779:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761048.108795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761048.108802:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761048/real 1789761048] req@ffff8eb645148e00 x1876699493907200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761048.108811:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493907200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761048.108816:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761048.108818:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761048.108821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761048.108838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761048.108850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761048.108856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761048/real 1789761048] req@ffff8eb64514a300 x1876699493907328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761048.108864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493907328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761048.108869:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761048.108871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761048.108873:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761048.108876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493907456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761048.108886:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761048.108891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761048/real 1789761048] req@ffff8eb64514b800 x1876699493907456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761048.108898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493907456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761048.108903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761048.108905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761048.108906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493907456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761048.812514:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761049.836553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761050.860523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761051.884546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761052.908521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761053.228451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761053.228459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761053.228463:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761053.228472:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1057 00000100:00080000:3.0:1789761053.228483:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761053.228491:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761053.228528:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493907584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761053.228551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761053.228554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761053.228555:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761053.228558:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1057 00000100:00080000:3.0:1789761053.228561:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761053.228564:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761053.228571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761053.228580:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493907712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761053.228589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761053.228592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761053.228593:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761053.228596:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1057 00000100:00080000:3.0:1789761053.228598:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761053.228606:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761053.228607:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493907584:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789761053.228610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761053.228619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493907840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761053.228629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761053.228632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761053.228634:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761053.228637:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1057 00000100:00080000:3.0:1789761053.228639:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761053.228642:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761053.228645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761053.228653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493907968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761053.228668:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761053.228701:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761053/real 1789761053] req@ffff8eb64514b100 x1876699493907584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761053.228717:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493907584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761053.228725:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761053.228728:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761053.228731:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493907584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761053.228737:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761053.228753:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761053.228763:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761053/real 1789761053] req@ffff8eb645149880 x1876699493907712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761053.228771:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493907712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761053.228776:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761053.228779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761053.228781:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761053.228798:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761053.228809:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761053.228815:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761053/real 1789761053] req@ffff8eb645148000 x1876699493907840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761053.228827:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493907840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761053.228832:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761053.228834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761053.228836:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493907840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761053.228839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493907968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761053.228848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761053.228853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761053/real 1789761053] req@ffff8eb64514bb80 x1876699493907968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761053.228861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493907968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761053.228866:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761053.228867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761053.228869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493907968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761053.932561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761054.956521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761055.980572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761057.004597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761058.028523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761058.348491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761058.348500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761058.348504:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761058.348513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1062 00000100:00080000:3.0:1789761058.348522:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761058.348528:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761058.348559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493908096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761058.348582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761058.348586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761058.348588:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761058.348592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1062 00000100:00080000:3.0:1789761058.348596:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761058.348600:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761058.348616:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761058.348629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493908224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761058.348630:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493908096:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761058.348640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761058.348644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761058.348645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761058.348648:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1062 00000100:00080000:3.0:1789761058.348651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761058.348655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761058.348660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761058.348670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493908352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761058.348679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761058.348682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761058.348684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789761058.348687:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761058.348697:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1062 00000100:00080000:3.0:1789761058.348701:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761058.348704:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761058.348709:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761058.348718:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493908480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761058.348724:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761058/real 1789761058] req@ffff8eb64514ad80 x1876699493908096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761058.348742:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493908096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761058.348753:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761058.348757:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761058.348761:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493908096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761058.348783:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761058.348803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761058.348812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761058/real 1789761058] req@ffff8eb64514b480 x1876699493908224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761058.348821:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493908224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761058.348826:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761058.348829:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761058.348831:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761058.348835:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908352:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761058.348846:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761058.348852:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761058/real 1789761058] req@ffff8eb645148700 x1876699493908352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761058.348860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493908352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761058.348865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761058.348867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761058.348869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761058.348881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493908480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761058.348896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761058.348903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761058/real 1789761058] req@ffff8eb645149180 x1876699493908480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761058.348910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493908480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761058.348915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761058.348917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761058.348920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493908480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761059.052577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761060.076523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761061.100583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761062.124594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761063.148555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761063.468472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761063.468480:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761063.468483:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761063.468491:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1067 00000100:00080000:0.0:1789761063.468500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761063.468506:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761063.468546:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493908608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761063.468566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761063.468569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761063.468570:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761063.468573:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1067 00000100:00080000:0.0:1789761063.468575:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761063.468578:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761063.468585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761063.468594:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493908736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761063.468601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761063.468604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761063.468605:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761063.468608:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1067 00000100:00080000:0.0:1789761063.468609:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761063.468612:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761063.468615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761063.468618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493908608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761063.468625:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493908864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761063.468633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761063.468635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761063.468636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761063.468640:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1067 00000100:00080000:0.0:1789761063.468642:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761063.468644:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761063.468648:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761063.468655:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493908992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761063.468677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761063.468720:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761063/real 1789761063] req@ffff8eb64304e680 x1876699493908608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761063.468737:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493908608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761063.468746:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761063.468749:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761063.468752:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493908608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761063.468758:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761063.468773:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761063.468782:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761063/real 1789761063] req@ffff8eb64304f480 x1876699493908736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761063.468790:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493908736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761063.468796:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761063.468798:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761063.468800:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761063.468818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908864:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761063.468830:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761063.468835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761063/real 1789761063] req@ffff8eb64304d880 x1876699493908864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761063.468843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493908864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761063.468848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761063.468850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761063.468851:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493908864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761063.468855:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493908992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761063.468864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761063.468870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761063/real 1789761063] req@ffff8eb64304d500 x1876699493908992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761063.468877:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493908992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761063.468883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761063.468884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761063.468887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493908992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761064.172555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761065.196542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761066.220504:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761067.244535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761068.268553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761068.588556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761068.588569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761068.588574:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761068.588588:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1073 00000100:00080000:3.0:1789761068.588603:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761068.588612:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761068.588663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493909120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761068.588694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761068.588698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761068.588700:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761068.588705:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1073 00000100:00080000:3.0:1789761068.588710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761068.588715:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761068.588729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761068.588741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493909120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761068.588747:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493909248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761068.588761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761068.588765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761068.588767:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761068.588771:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1073 00000100:00080000:3.0:1789761068.588776:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761068.588781:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761068.588788:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761068.588802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493909376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761068.588814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761068.588816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761068.588822:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761068.588824:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761068.588830:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1073 00000100:00080000:3.0:1789761068.588835:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761068.588840:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761068.588846:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761068.588857:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493909504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761068.588866:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761068/real 1789761068] req@ffff8eb645149180 x1876699493909120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761068.588901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493909120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761068.588911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761068.588915:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761068.588919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493909120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761068.588942:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761068.588961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761068.588971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761068/real 1789761068] req@ffff8eb645148700 x1876699493909248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761068.588979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493909248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761068.588985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761068.588988:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761068.588991:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761068.588995:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761068.589008:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761068.589014:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761068/real 1789761068] req@ffff8eb64514b480 x1876699493909376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761068.589022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493909376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761068.589027:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761068.589029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761068.589031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761068.589035:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493909504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761068.589046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761068.589052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761068/real 1789761068] req@ffff8eb64514ad80 x1876699493909504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761068.589059:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493909504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761068.589064:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761068.589066:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761068.589068:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493909504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761069.292585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761070.316526:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761071.340560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761072.364573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761073.388534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761073.708523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761073.708531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761073.708534:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761073.708544:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1078 00000100:00080000:0.0:1789761073.708564:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761073.708571:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761073.708604:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493909632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761073.708634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761073.708638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761073.708640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761073.708645:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1078 00000100:00080000:0.0:1789761073.708649:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761073.708653:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761073.708663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761073.708674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493909760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761073.708683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761073.708688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761073.708689:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493909632:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761073.708690:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761073.708698:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1078 00000100:00080000:0.0:1789761073.708702:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761073.708705:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761073.708708:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761073.708717:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493909888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761073.708724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761073.708727:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761073.708728:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761073.708730:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1078 00000100:00080000:0.0:1789761073.708733:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761073.708735:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761073.708738:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761073.708744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493910016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761073.708810:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761073.708860:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761073.708891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493910016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761074.412507:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761075.436524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761076.460566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761076.780607:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789761076.780645:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761076.780659:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761076.780688:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761076.780693:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761076.780699:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789761076.780756:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761073/real 1789761076] req@ffff8eb64304ce00 x1876699493909760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.780795:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493909760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.780810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.780816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761076.780821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761076.780845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761073/real 1789761076] req@ffff8eb64304dc00 x1876699493909888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.780860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493909888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.780868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.780880:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761076.780887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493909888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761076.780896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761073/real 1789761076] req@ffff8eb64304c380 x1876699493910016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.780910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493910016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.780918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.780922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761076.780926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493910016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761076.780934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761073/real 1789761076] req@ffff8eb64304f800 x1876699493909632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761076.780948:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493909632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761076.780956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761076.780959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761076.780963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493909632:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761077.484595:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761078.508580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761078.828517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761078.828531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761078.828536:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761078.828550:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1083 00000100:00080000:3.0:1789761078.828564:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761078.828572:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761078.828613:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493910144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761078.828645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761078.828650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761078.828652:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761078.828657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1083 00000100:00080000:3.0:1789761078.828662:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761078.828667:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761078.828679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761078.828691:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493910144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761078.828697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493910272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761078.828711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761078.828716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761078.828718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761078.828722:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1083 00000100:00080000:3.0:1789761078.828726:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761078.828729:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761078.828735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761078.828745:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493910400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761078.828753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761078.828756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761078.828757:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761078.828761:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1083 00000100:00080000:3.0:1789761078.828764:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761078.828767:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761078.828771:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761078.828776:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761078.828794:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493910528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761078.828834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761078/real 1789761078] req@ffff8eb64514bb80 x1876699493910144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761078.828860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493910144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761078.828882:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761078.828888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761078.828893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493910144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761078.828927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761078.828958:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761078.828974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761078/real 1789761078] req@ffff8eb645148000 x1876699493910272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761078.828989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493910272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761078.828999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761078.829003:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761078.829007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761078.829014:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761078.829041:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761078.829054:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761078/real 1789761078] req@ffff8eb645149880 x1876699493910400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761078.829067:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493910400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761078.829076:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761078.829079:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761078.829084:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761078.829090:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493910528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761078.829111:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761078.829123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761078/real 1789761078] req@ffff8eb64514b100 x1876699493910528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761078.829136:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493910528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761078.829145:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761078.829148:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761078.829152:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493910528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761079.532534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761080.556558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761081.580590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761082.604606:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761083.628534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761083.948494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761083.948502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761083.948506:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761083.948515:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1088 00000100:00080000:3.0:1789761083.948524:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761083.948530:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761083.948565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493910656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761083.948595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761083.948599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761083.948601:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761083.948606:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1088 00000100:00080000:3.0:1789761083.948610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761083.948614:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761083.948623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761083.948646:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493910784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761083.948650:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493910656:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761083.948667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761083.948670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761083.948672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761083.948676:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1088 00000100:00080000:3.0:1789761083.948680:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761083.948683:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761083.948689:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761083.948702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493910912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761083.948713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761083.948716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761083.948721:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761083.948721:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761083.948730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1088 00000100:00080000:3.0:1789761083.948734:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761083.948740:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761083.948744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761083.948755:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493911040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761083.948765:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761083/real 1789761083] req@ffff8eb64514b800 x1876699493910656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761083.948800:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493910656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761083.948811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761083.948815:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761083.948819:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493910656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761083.948839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761083.948862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761083.948874:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761083/real 1789761083] req@ffff8eb64514a300 x1876699493910784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761083.948887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493910784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761083.948896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761083.948899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761083.948903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761083.948909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910912:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761083.948927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761083.948938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761083/real 1789761083] req@ffff8eb645148e00 x1876699493910912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761083.948950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493910912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761083.948958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761083.948961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761083.948964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493910912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761083.948971:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493911040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761083.948999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761083.949011:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761083/real 1789761083] req@ffff8eb645148380 x1876699493911040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761083.949022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493911040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761083.949030:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761083.949033:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761083.949037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493911040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761084.652510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761085.676530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761086.700558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761087.724510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761088.748503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761089.068457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761089.068468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761089.068473:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761089.068483:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1093 00000100:00080000:3.0:1789761089.068494:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761089.068502:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761089.068535:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493911168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761089.068557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761089.068560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761089.068562:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761089.068566:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1093 00000100:00080000:3.0:1789761089.068571:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761089.068575:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761089.068584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761089.068598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493911296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761089.068610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761089.068614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761089.068615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761089.068621:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1093 00000100:00100000:1.0:1789761089.068623:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493911168:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761089.068625:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761089.068632:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761089.068638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761089.068650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493911424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761089.068659:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761089.068662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761089.068663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761089.068666:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1093 00000100:00080000:3.0:1789761089.068668:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761089.068671:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761089.068674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761089.068681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493911552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761089.068687:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761089.068725:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761089/real 1789761089] req@ffff8eb645149c00 x1876699493911168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761089.068759:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493911168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761089.068771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761089.068775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761089.068780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493911168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761089.068787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761089.068811:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761089.068826:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761089/real 1789761089] req@ffff8eb645149500 x1876699493911296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761089.068835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493911296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761089.068841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761089.068843:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761089.068845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761089.068862:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761089.068876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761089.068882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761089/real 1789761089] req@ffff8eb645148a80 x1876699493911424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761089.068890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493911424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761089.068895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761089.068897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761089.068899:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761089.068902:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493911552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761089.068911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761089.068917:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761089/real 1789761089] req@ffff8eb64514a680 x1876699493911552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761089.068924:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493911552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761089.068929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761089.068931:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761089.068932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493911552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761089.772513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761090.796539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761091.820553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761092.844581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761093.868506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761094.188475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761094.188484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761094.188488:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761094.188496:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1098 00000100:00080000:3.0:1789761094.188505:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761094.188511:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761094.188543:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493911680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761094.188571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761094.188574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761094.188575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761094.188578:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1098 00000100:00080000:3.0:1789761094.188581:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761094.188583:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761094.188590:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761094.188600:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493911808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761094.188607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761094.188609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761094.188610:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761094.188612:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1098 00000100:00080000:3.0:1789761094.188614:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761094.188618:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761094.188622:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493911680:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789761094.188623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761094.188633:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493911936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761094.188647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761094.188650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761094.188651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761094.188657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1098 00000100:00080000:3.0:1789761094.188661:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761094.188664:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761094.188667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761094.188676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493912064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761094.188686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761094.188733:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761094/real 1789761094] req@ffff8eb64514a680 x1876699493911680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761094.188755:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493911680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761094.188763:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761094.188765:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761094.188768:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493911680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761094.188773:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761094.188790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761094.188798:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761094/real 1789761094] req@ffff8eb645148a80 x1876699493911808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761094.188806:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493911808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761094.188812:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761094.188814:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761094.188816:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761094.188834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761094.188845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761094.188851:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761094/real 1789761094] req@ffff8eb645149500 x1876699493911936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761094.188859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493911936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761094.188868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761094.188878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761094.188883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493911936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761094.188886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493912064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761094.188898:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761094.188904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761094/real 1789761094] req@ffff8eb645149c00 x1876699493912064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761094.188912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493912064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761094.188917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761094.188919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761094.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493912064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761094.892554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761095.916610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761096.940528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761097.964521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761098.988531:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761099.308597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761099.308608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761099.308612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761099.308624:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1103 00000100:00080000:0.0:1789761099.308637:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761099.308644:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761099.308702:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493912192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761099.308731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761099.308734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761099.308735:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761099.308742:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1103 00000100:00080000:0.0:1789761099.308745:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761099.308748:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761099.308759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761099.308768:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493912320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761099.308777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761099.308780:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761099.308782:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761099.308786:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1103 00000100:00100000:1.0:1789761099.308788:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493912192:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761099.308789:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761099.308796:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761099.308800:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761099.308808:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493912448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761099.308815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761099.308818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761099.308819:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761099.308822:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1103 00000100:00080000:0.0:1789761099.308824:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761099.308827:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761099.308829:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761099.308836:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493912576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761099.308865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761099.308914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761099/real 1789761099] req@ffff8eb64304c000 x1876699493912192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761099.308942:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493912192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761099.308954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761099.308960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761099.308964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493912192:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761099.308991:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761099.309010:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761099.309020:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761099/real 1789761099] req@ffff8eb64304d180 x1876699493912320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761099.309029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493912320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761099.309035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761099.309037:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761099.309040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761099.309043:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912448:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761099.309055:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761099.309060:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761099/real 1789761099] req@ffff8eb64304ed80 x1876699493912448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761099.309067:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493912448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761099.309073:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761099.309075:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761099.309076:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761099.309080:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493912576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761099.309091:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761099.309097:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761099/real 1789761099] req@ffff8eb64304e300 x1876699493912576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761099.309104:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493912576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761099.309110:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761099.309111:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761099.309113:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493912576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761100.012529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761101.036527:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761102.060546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761103.084556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761104.108542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761104.428476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761104.428487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761104.428492:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761104.428501:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1108 00000100:00080000:0.0:1789761104.428511:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761104.428518:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761104.428558:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493912704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761104.428589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761104.428593:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761104.428595:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761104.428600:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1108 00000100:00080000:0.0:1789761104.428605:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761104.428610:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761104.428633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761104.428641:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493912704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761104.428651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493912832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761104.428665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761104.428669:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761104.428671:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761104.428676:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1108 00000100:00080000:0.0:1789761104.428683:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761104.428688:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761104.428694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761104.428695:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761104.428711:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493912960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761104.428724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761104.428728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761104.428730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761104.428734:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761104/real 1789761104] req@ffff8eb64304e300 x1876699493912704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761104.428738:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1108 00000100:00080000:0.0:1789761104.428746:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761104.428752:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761104.428757:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493912704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761104.428761:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761104.428770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761104.428775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761104.428779:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493913088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761104.428782:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493912704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761104.428808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761104.428825:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761104.428834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761104/real 1789761104] req@ffff8eb64304ed80 x1876699493912832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761104.428845:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493912832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761104.428852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761104.428855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761104.428857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761104.428861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761104.428875:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761104.428883:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761104/real 1789761104] req@ffff8eb64304d180 x1876699493912960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761104.428891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493912960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761104.428896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761104.428898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761104.428899:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493912960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761104.428903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493913088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761104.428915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761104.428924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761104/real 1789761104] req@ffff8eb64304c000 x1876699493913088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761104.428932:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493913088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761104.428942:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761104.428944:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761104.428945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493913088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761105.132503:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761106.156566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761107.180524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761108.204514:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761109.228562:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761109.548518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761109.548526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761109.548529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761109.548537:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1114 00000100:00080000:3.0:1789761109.548545:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761109.548551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761109.548599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493913216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761109.548636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761109.548639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761109.548642:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761109.548646:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1114 00000100:00080000:3.0:1789761109.548649:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761109.548653:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761109.548661:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761109.548676:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493913216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761109.548677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493913344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761109.548691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761109.548693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761109.548694:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761109.548697:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1114 00000100:00080000:3.0:1789761109.548700:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761109.548702:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761109.548706:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761109.548727:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493913472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761109.548734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761109.548737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761109.548738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761109.548740:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1114 00000100:00080000:3.0:1789761109.548743:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761109.548747:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761109.548750:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761109.548754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761109.548763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493913600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761109.548796:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761109/real 1789761109] req@ffff8eb645148380 x1876699493913216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761109.548809:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493913216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761109.548817:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761109.548819:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761109.548822:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493913216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761109.548840:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761109.548882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761109.548892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761109/real 1789761109] req@ffff8eb645148e00 x1876699493913344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761109.548901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493913344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761109.548908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761109.548926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761109.548929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761109.548933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913472:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761109.548946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761109.548951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761109/real 1789761109] req@ffff8eb64514a300 x1876699493913472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761109.548959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493913472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761109.548975:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761109.548977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761109.548979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761109.548982:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493913600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761109.549009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761109.549016:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761109/real 1789761109] req@ffff8eb64514b800 x1876699493913600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761109.549024:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493913600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761109.549030:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761109.549032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761109.549034:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493913600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761110.252561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761111.276518:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761112.300584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761113.324523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761114.348528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761114.668495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761114.668504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761114.668507:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761114.668515:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1119 00000100:00080000:1.0:1789761114.668524:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761114.668530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761114.668571:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493913728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761114.668615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761114.668617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761114.668619:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761114.668622:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1119 00000100:00080000:1.0:1789761114.668625:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761114.668628:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761114.668635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761114.668645:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493913856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761114.668653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761114.668657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761114.668658:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493913728:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761114.668659:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761114.668667:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1119 00000100:00080000:1.0:1789761114.668670:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761114.668672:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761114.668676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761114.668686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493913984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761114.668694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761114.668696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761114.668697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761114.668700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1119 00000100:00080000:1.0:1789761114.668710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761114.668713:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761114.668716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761114.668721:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761114.668727:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493914112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761114.668765:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761114/real 1789761114] req@ffff8eb65016d500 x1876699493913728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761114.668779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493913728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761114.668804:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761114.668807:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761114.668810:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493913728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761114.668829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761114.668847:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761114.668855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761114/real 1789761114] req@ffff8eb65016fb80 x1876699493913856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761114.668863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493913856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761114.668868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761114.668871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761114.668873:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761114.668876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761114.668887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761114.668893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761114/real 1789761114] req@ffff8eb65016e300 x1876699493913984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761114.668900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493913984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761114.668905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761114.668907:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761114.668909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493913984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761114.668913:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493914112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761114.668926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761114.668936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761114/real 1789761114] req@ffff8eb65016dc00 x1876699493914112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761114.668981:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493914112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761114.668994:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761114.668997:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761114.669000:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493914112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761115.373481:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761116.396599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761117.420593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761118.444559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761119.468586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761119.788498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761119.788517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761119.788521:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761119.788529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1124 00000100:00080000:0.0:1789761119.788538:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761119.788544:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761119.788592:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493914240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761119.788619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761119.788622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761119.788623:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761119.788626:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1124 00000100:00080000:0.0:1789761119.788629:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761119.788632:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761119.788640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761119.788651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493914368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761119.788661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761119.788662:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493914240:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761119.788664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761119.788667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761119.788674:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1124 00000100:00080000:0.0:1789761119.788676:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761119.788678:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761119.788681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761119.788688:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493914496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761119.788694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761119.788697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761119.788699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761119.788701:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1124 00000100:00080000:0.0:1789761119.788704:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761119.788706:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761119.788709:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761119.788715:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493914624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761119.788729:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761119.788772:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761119/real 1789761119] req@ffff8eb64304f800 x1876699493914240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761119.788787:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493914240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761119.788795:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761119.788799:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761119.788804:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493914240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761119.788824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761119.788840:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761119.788848:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761119/real 1789761119] req@ffff8eb64304c380 x1876699493914368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761119.788857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493914368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761119.788862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761119.788865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761119.788867:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761119.788870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761119.788882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761119.788888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761119/real 1789761119] req@ffff8eb64304dc00 x1876699493914496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761119.788895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493914496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761119.788900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761119.788902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761119.788904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761119.788908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493914624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761119.788918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761119.788924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761119/real 1789761119] req@ffff8eb64304ce00 x1876699493914624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761119.788935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493914624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761119.788940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761119.788942:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761119.788944:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493914624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761120.492574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761121.516576:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761122.540543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761123.564588:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761124.588544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761124.908554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761124.908566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761124.908570:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761124.908581:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1129 00000100:00080000:3.0:1789761124.908592:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761124.908598:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761124.908628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493914752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761124.908652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761124.908658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761124.908659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761124.908663:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1129 00000100:00080000:3.0:1789761124.908665:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761124.908667:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761124.908675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761124.908686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493914880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761124.908697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761124.908703:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493914752:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761124.908704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761124.908707:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761124.908716:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1129 00000100:00080000:3.0:1789761124.908720:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761124.908725:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761124.908731:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761124.908745:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493915008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761124.908753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761124.908756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761124.908758:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761124.908761:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1129 00000100:00080000:3.0:1789761124.908763:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761124.908768:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761124.908769:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761124.908777:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761124.908789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493915136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761124.908812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761124/real 1789761124] req@ffff8eb64514b100 x1876699493914752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761124.908827:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493914752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761124.908835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761124.908838:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761124.908841:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493914752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761124.908859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761124.908874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761124.908882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761124/real 1789761124] req@ffff8eb645149880 x1876699493914880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761124.908890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493914880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761124.908901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761124.908903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761124.908905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493914880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761124.908909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915008:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761124.908919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761124.908925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761124/real 1789761124] req@ffff8eb645148000 x1876699493915008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761124.908932:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493915008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761124.908938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761124.908939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761124.908941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761124.908945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493915136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761124.908955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761124.908961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761124/real 1789761124] req@ffff8eb64514bb80 x1876699493915136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761124.908968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493915136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761124.908973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761124.908974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761124.908977:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493915136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761125.612668:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761126.636575:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761127.660523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761128.684544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761129.708545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761130.028527:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761130.028537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761130.028541:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761130.028551:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1134 00000100:00080000:0.0:1789761130.028561:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761130.028569:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761130.028606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493915264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761130.028645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761130.028649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761130.028651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761130.028655:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1134 00000100:00080000:0.0:1789761130.028659:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761130.028663:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761130.028673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761130.028687:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493915264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761130.028693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493915392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761130.028706:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761130.028710:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761130.028712:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761130.028716:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1134 00000100:00080000:0.0:1789761130.028729:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761130.028734:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761130.028739:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761130.028740:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761130.028754:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493915520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761130.028767:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761130.028770:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761130.028772:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761130.028778:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1134 00000100:00080000:0.0:1789761130.028781:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761130.028786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761130/real 1789761130] req@ffff8eb64304d500 x1876699493915264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761130.028789:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761130.028799:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761130.028808:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493915264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761130.028814:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493915648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761130.028819:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761130.028828:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761130.028834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493915264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761130.028857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761130.028880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761130.028892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761130/real 1789761130] req@ffff8eb64304d880 x1876699493915392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761130.028906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493915392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761130.028915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761130.028918:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761130.028920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761130.028924:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761130.028941:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761130.028949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761130/real 1789761130] req@ffff8eb64304f480 x1876699493915520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761130.028957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493915520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761130.028962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761130.028965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761130.028967:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761130.028970:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493915648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761130.028985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761130.028996:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761130/real 1789761130] req@ffff8eb64304e680 x1876699493915648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761130.029009:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493915648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761130.029017:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761130.029021:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761130.029024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493915648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761130.732523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761131.756609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761132.780515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761133.805486:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761134.828830:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761135.148500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761135.148508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761135.148511:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761135.148520:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1139 00000100:00080000:1.0:1789761135.148530:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761135.148536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761135.148566:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493915776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761135.148587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761135.148589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761135.148591:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761135.148594:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1139 00000100:00080000:1.0:1789761135.148596:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761135.148600:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761135.148608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761135.148616:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493915776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761135.148619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493915904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761135.148630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761135.148632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761135.148633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761135.148635:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1139 00000100:00080000:1.0:1789761135.148637:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761135.148640:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761135.148643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761135.148653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493916032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761135.148664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761135.148667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761135.148668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761135.148671:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1139 00000100:00080000:1.0:1789761135.148673:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761135.148677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761135.148680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761135.148685:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761135.148696:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493916160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761135.148715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761135/real 1789761135] req@ffff8eb65016ce00 x1876699493915776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761135.148732:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493915776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761135.148741:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761135.148745:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761135.148748:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493915776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761135.148766:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761135.148780:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761135.148788:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761135/real 1789761135] req@ffff8eb65016c700 x1876699493915904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761135.148796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493915904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761135.148802:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761135.148804:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761135.148806:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493915904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761135.148810:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761135.148820:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761135.148826:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761135/real 1789761135] req@ffff8eb65016df80 x1876699493916032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761135.148833:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493916032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761135.148838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761135.148839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761135.148841:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761135.148845:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493916160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761135.148854:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761135.148859:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761135/real 1789761135] req@ffff8eb65016c000 x1876699493916160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761135.148866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493916160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761135.148879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761135.148881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761135.148886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493916160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761135.852551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761136.876541:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761137.900539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761138.924595:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761139.948504:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761140.268479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761140.268487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761140.268490:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761140.268499:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1144 00000100:00080000:0.0:1789761140.268508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761140.268514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761140.268542:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493916288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761140.268564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761140.268567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761140.268568:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761140.268571:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1144 00000100:00080000:0.0:1789761140.268573:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761140.268577:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761140.268584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761140.268593:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493916416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761140.268601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761140.268604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761140.268608:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761140.268609:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493916288:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761140.268612:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1144 00000100:00080000:0.0:1789761140.268616:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761140.268620:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761140.268623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761140.268630:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493916544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761140.268637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761140.268640:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761140.268642:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761140.268645:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1144 00000100:00080000:0.0:1789761140.268659:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761140.268662:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761140.268665:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761140.268672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493916672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761140.268714:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761140.268749:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761140.268772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493916672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761140.972531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761141.996534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761143.020550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761143.340610:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789761143.340648:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761143.340665:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789761143.340699:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789761143.340705:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789761143.340710:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789761143.340750:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761140/real 1789761143] req@ffff8eb64304ca80 x1876699493916416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.340778:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493916416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.340796:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.340802:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761143.340808:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761143.340816:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761140/real 1789761143] req@ffff8eb64304ea00 x1876699493916544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.340843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493916544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.340852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.340856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761143.340860:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761143.340867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761140/real 1789761143] req@ffff8eb64304c700 x1876699493916672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.340880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493916672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.340887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.340890:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761143.340894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493916672:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761143.340901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761140/real 1789761143] req@ffff8eb64304f100 x1876699493916288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.340912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493916288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.340920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.340936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761143.340940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493916288:192.168.203.160@tcp:250:kworker.0 00010000:00010000:1.0:1789761144.044520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761145.068558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761145.388474:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761145.388484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761145.388489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761145.388500:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1149 00000100:00080000:1.0:1789761145.388512:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761145.388520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761145.388555:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493916800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761145.388582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761145.388585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761145.388588:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761145.388592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1149 00000100:00080000:1.0:1789761145.388597:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761145.388602:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761145.388611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761145.388625:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493916800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761145.388626:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493916928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761145.388643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761145.388647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761145.388649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761145.388653:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1149 00000100:00080000:1.0:1789761145.388657:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761145.388662:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761145.388667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761145.388680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493917056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761145.388691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761145.388696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789761145.388697:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761145.388700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761145.388712:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1149 00000100:00080000:1.0:1789761145.388717:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761145.388722:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761145.388727:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761145.388739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493917184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761145.388746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761145/real 1789761145] req@ffff8eb65016f480 x1876699493916800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761145.388779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493916800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761145.388790:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761145.388794:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761145.388799:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493916800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761145.388823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761145.388848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761145.388861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761145/real 1789761145] req@ffff8eb65016ed80 x1876699493916928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761145.388875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493916928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761145.388884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761145.388890:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761145.388893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493916928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761145.388899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761145.388919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761145.388931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761145/real 1789761145] req@ffff8eb65016d180 x1876699493917056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761145.388943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493917056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761145.388957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761145.388960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761145.388963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761145.388969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493917184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761145.388990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761145.389001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761145/real 1789761145] req@ffff8eb65016f100 x1876699493917184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761145.389012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493917184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761145.389020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761145.389024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761145.389027:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493917184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761146.092505:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761147.116555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761148.140519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761149.164571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761150.188545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761150.508482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761150.508489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761150.508492:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761150.508500:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1155 00000100:00080000:1.0:1789761150.508508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761150.508514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761150.508544:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493917312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761150.508566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761150.508570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761150.508571:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761150.508574:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1155 00000100:00080000:1.0:1789761150.508576:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761150.508578:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761150.508585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761150.508600:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493917440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761150.508609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761150.508612:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493917312:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761150.508613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761150.508616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761150.508624:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1155 00000100:00080000:1.0:1789761150.508626:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761150.508628:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761150.508632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761150.508641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493917568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761150.508648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761150.508651:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761150.508652:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761150.508655:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1155 00000100:00080000:1.0:1789761150.508664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761150.508667:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761150.508670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761150.508675:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761150.508679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493917696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761150.508716:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761150/real 1789761150] req@ffff8eb65016ca80 x1876699493917312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761150.508732:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493917312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761150.508740:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761150.508744:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761150.508747:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493917312:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761150.508766:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761150.508781:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761150.508790:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761150/real 1789761150] req@ffff8eb65016f100 x1876699493917440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761150.508798:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493917440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761150.508804:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761150.508806:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761150.508808:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761150.508812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917568:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761150.508822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761150.508828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761150/real 1789761150] req@ffff8eb65016d180 x1876699493917568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761150.508835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493917568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761150.508848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761150.508850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761150.508852:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761150.508856:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493917696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761150.508868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761150.508875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761150/real 1789761150] req@ffff8eb65016ed80 x1876699493917696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761150.508882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493917696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761150.508887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761150.508889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761150.508890:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493917696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761151.212572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761152.236554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761153.260540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761154.284585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761155.308564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761155.628567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761155.628577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761155.628582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761155.628590:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1160 00000100:00080000:0.0:1789761155.628599:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761155.628605:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761155.628647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493917824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761155.628679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761155.628682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761155.628683:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761155.628687:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1160 00000100:00080000:0.0:1789761155.628691:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761155.628695:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761155.628702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761155.628711:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493917824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761155.628716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493917952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761155.628726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761155.628728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761155.628730:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761155.628733:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1160 00000100:00080000:0.0:1789761155.628736:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761155.628740:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761155.628745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761155.628759:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493918080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761155.628772:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761155.628777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789761155.628779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761155.628782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761155.628788:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1160 00000100:00080000:0.0:1789761155.628792:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761155.628796:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761155.628801:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761155.628812:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493918208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761155.628838:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761155/real 1789761155] req@ffff8eb64304f100 x1876699493917824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761155.628856:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493917824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761155.628868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761155.628872:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761155.628875:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493917824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761155.628895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761155.628912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761155.628921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761155/real 1789761155] req@ffff8eb64304c700 x1876699493917952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761155.628929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493917952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761155.628935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761155.628937:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761155.628939:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493917952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761155.628943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761155.628954:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761155.628960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761155/real 1789761155] req@ffff8eb64304ea00 x1876699493918080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761155.628968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493918080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761155.628973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761155.628975:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761155.628976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761155.628980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493918208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761155.628990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761155.628998:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761155/real 1789761155] req@ffff8eb64304ca80 x1876699493918208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761155.629010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493918208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761155.629019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761155.629022:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761155.629026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493918208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761156.332563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761157.356495:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761158.380559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761159.404505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761160.428553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761160.748476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761160.748487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761160.748490:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761160.748498:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1165 00000100:00080000:1.0:1789761160.748507:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761160.748512:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761160.748544:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493918336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761160.748571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761160.748575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761160.748577:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761160.748581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1165 00000100:00080000:1.0:1789761160.748585:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761160.748590:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761160.748600:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761160.748614:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493918336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761160.748617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493918464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761160.748633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761160.748636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761160.748637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761160.748640:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1165 00000100:00080000:1.0:1789761160.748645:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761160.748649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761160.748654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761160.748666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493918592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761160.748677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761160.748681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761160.748686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761160.748688:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761160.748694:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1165 00000100:00080000:1.0:1789761160.748698:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761160.748702:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761160.748706:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761160.748719:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493918720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761160.748722:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761160/real 1789761160] req@ffff8eb65016f480 x1876699493918336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761160.748752:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493918336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761160.748764:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761160.748768:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761160.748773:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493918336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761160.748797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761160.748818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761160.748828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761160/real 1789761160] req@ffff8eb65016c000 x1876699493918464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761160.748844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493918464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761160.748850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761160.748853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761160.748855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918464:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761160.748859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918592:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761160.748872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761160.748879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761160/real 1789761160] req@ffff8eb65016df80 x1876699493918592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761160.748891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493918592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761160.748896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761160.748898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761160.748901:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918592:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761160.748904:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493918720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761160.748914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761160.748919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761160/real 1789761160] req@ffff8eb65016c700 x1876699493918720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761160.748927:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493918720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761160.748932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761160.748933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761160.748935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493918720:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761161.452525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761162.476511:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761163.500509:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761164.524564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761165.548776:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761165.868470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761165.868479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761165.868483:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761165.868493:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1170 00000100:00080000:1.0:1789761165.868503:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761165.868510:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761165.868548:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493918848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761165.868574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761165.868578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761165.868580:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761165.868584:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1170 00000100:00080000:1.0:1789761165.868588:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761165.868593:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761165.868603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761165.868610:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493918848:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761165.868618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493918976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761165.868639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761165.868643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761165.868645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761165.868649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1170 00000100:00080000:1.0:1789761165.868653:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761165.868658:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761165.868664:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761165.868664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761165.868681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493919104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761165.868692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761165.868696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761165.868699:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761165/real 1789761165] req@ffff8eb65016ce00 x1876699493918848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761165.868700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761165.868710:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1170 00000100:00080000:1.0:1789761165.868716:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761165.868723:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493918848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761165.868724:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761165.868733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761165.868737:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761165.868742:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761165.868746:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493918848:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761165.868749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493919232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761165.868770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761165.868788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761165.868797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761165/real 1789761165] req@ffff8eb65016dc00 x1876699493918976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761165.868804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493918976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761165.868810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761165.868812:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761165.868814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493918976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761165.868817:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919104:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761165.868827:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761165.868833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761165/real 1789761165] req@ffff8eb65016e300 x1876699493919104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761165.868840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493919104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761165.868845:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761165.868847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761165.868848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761165.868857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493919232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761165.868867:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761165.868873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761165/real 1789761165] req@ffff8eb65016fb80 x1876699493919232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761165.868880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493919232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761165.868885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761165.868887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761165.868889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493919232:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761166.572552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761167.596499:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761168.620594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761169.644499:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761170.668524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761170.988434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761170.988442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761170.988446:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761170.988453:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1175 00000100:00080000:1.0:1789761170.988462:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761170.988467:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761170.988497:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493919360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761170.988518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761170.988521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761170.988522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761170.988525:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1175 00000100:00080000:1.0:1789761170.988527:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761170.988530:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761170.988537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761170.988546:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493919488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761170.988554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761170.988557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761170.988559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761170.988565:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493919360:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761170.988565:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1175 00000100:00080000:1.0:1789761170.988569:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761170.988576:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761170.988579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761170.988587:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493919616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761170.988594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761170.988597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761170.988598:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761170.988600:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1175 00000100:00080000:1.0:1789761170.988602:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761170.988604:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761170.988608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761170.988615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493919744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761170.988638:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761170.988670:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761170/real 1789761170] req@ffff8eb65016d500 x1876699493919360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761170.988683:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493919360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761170.988691:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761170.988694:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761170.988697:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493919360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761170.988717:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761170.988744:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761170.988758:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761170/real 1789761170] req@ffff8eb65016c380 x1876699493919488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761170.988772:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493919488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761170.988781:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761170.988784:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761170.988787:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919488:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761170.988792:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919616:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761170.988808:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761170.988816:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761170/real 1789761170] req@ffff8eb65016f800 x1876699493919616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761170.988824:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493919616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761170.988829:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761170.988831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761170.988833:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493919616:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761170.988837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493919744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761170.988851:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761170.988861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761170/real 1789761170] req@ffff8eb65016e680 x1876699493919744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761170.988880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493919744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761170.988889:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761170.988893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761170.988896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493919744:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761171.692502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761172.716527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761173.740609:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761174.764526:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761175.788595:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761176.108506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761176.108527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761176.108532:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761176.108545:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1180 00000100:00080000:1.0:1789761176.108559:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761176.108568:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761176.108619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493919872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761176.108651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761176.108657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761176.108659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761176.108664:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1180 00000100:00080000:1.0:1789761176.108669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761176.108675:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761176.108687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761176.108706:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493920000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761176.108709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493919872:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761176.108739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761176.108745:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761176.108747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761176.108754:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1180 00000100:00080000:1.0:1789761176.108776:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761176.108782:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761176.108789:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761176.108794:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761176.108819:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493920128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761176.108828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761176/real 1789761176] req@ffff8eb65016d880 x1876699493919872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761176.108840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761176.108844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761176.108846:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761176.108851:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493919872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761176.108852:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1180 00000100:00080000:1.0:1789761176.108861:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761176.108871:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:0.0:1789761176.108872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789761176.108884:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761176.108885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761176.108895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493919872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761176.108899:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493920256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761176.108936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761176.108958:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761176.108969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761176/real 1789761176] req@ffff8eb65016ea00 x1876699493920000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761176.108977:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493920000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761176.108985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761176.108987:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761176.108990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761176.108994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920128:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761176.109006:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761176.109012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761176/real 1789761176] req@ffff8eb65016e680 x1876699493920128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761176.109020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493920128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761176.109025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761176.109027:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761176.109029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761176.109033:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493920256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761176.109044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761176.109049:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761176/real 1789761176] req@ffff8eb65016f800 x1876699493920256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761176.109056:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493920256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761176.109061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761176.109063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761176.109066:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493920256:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761176.812536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761177.836559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761178.860577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761179.884523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761180.908505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761181.228523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761181.228533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761181.228537:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761181.228547:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1185 00000100:00080000:0.0:1789761181.228566:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761181.228574:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761181.228610:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493920384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761181.228638:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761181.228643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761181.228645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761181.228649:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1185 00000100:00080000:0.0:1789761181.228653:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761181.228658:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761181.228668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761181.228676:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493920384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761181.228684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493920512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761181.228696:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761181.228700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761181.228702:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761181.228706:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1185 00000100:00080000:0.0:1789761181.228711:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761181.228715:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761181.228721:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761181.228734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493920640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761181.228738:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789761181.228750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761181.228754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761181.228756:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761181.228762:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1185 00000100:00080000:0.0:1789761181.228766:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761181.228772:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761181.228777:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761181.228780:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761181/real 1789761181] req@ffff8eb64304e680 x1876699493920384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761181.228793:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493920768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761181.228805:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493920384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761181.228821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761181.228826:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761181.228830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493920384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761181.228857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761181.228882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761181.228898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761181/real 1789761181] req@ffff8eb64304f480 x1876699493920512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761181.228911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493920512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761181.228920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761181.228924:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761181.228928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761181.228935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920640:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761181.228955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761181.228967:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761181/real 1789761181] req@ffff8eb64304d880 x1876699493920640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761181.228979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493920640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761181.228987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761181.228990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761181.228994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493920640:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761181.229000:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493920768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761181.229021:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761181.229032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761181/real 1789761181] req@ffff8eb64304d500 x1876699493920768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761181.229062:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493920768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761181.229072:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761181.229076:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761181.229079:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493920768:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761181.932587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761182.956585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761183.980611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761185.004543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761186.028616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761186.348503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761186.348518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761186.348523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761186.348538:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1190 00000100:00080000:0.0:1789761186.348552:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761186.348562:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761186.348611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493920896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761186.348639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761186.348644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761186.348646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761186.348651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1190 00000100:00080000:0.0:1789761186.348655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761186.348660:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761186.348672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761186.348687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493921024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761186.348698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761186.348699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493920896:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761186.348702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761186.348704:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761186.348708:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1190 00000100:00080000:0.0:1789761186.348712:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761186.348716:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761186.348720:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761186.348730:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493921152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761186.348741:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761186.348745:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761186.348746:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761186.348750:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1190 00000100:00080000:0.0:1789761186.348768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761186.348776:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761186.348782:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761186.348786:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761186.348793:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493921280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761186.348836:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761186/real 1789761186] req@ffff8eb64304ce00 x1876699493920896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761186.348855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493920896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761186.348864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761186.348867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761186.348870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493920896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761186.348893:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761186.348912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761186.348921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761186/real 1789761186] req@ffff8eb64304dc00 x1876699493921024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761186.348930:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493921024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761186.348936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761186.348938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761186.348940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921024:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761186.348943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921152:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761186.348963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761186.348971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761186/real 1789761186] req@ffff8eb64304c380 x1876699493921152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761186.348983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493921152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761186.348989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761186.348991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761186.348993:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921152:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761186.348996:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493921280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761186.349028:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761186.349037:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761186/real 1789761186] req@ffff8eb64304f800 x1876699493921280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761186.349045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493921280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761186.349051:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761186.349053:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761186.349055:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493921280:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761187.052565:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761188.076603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761189.100606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761190.124637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761191.148561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761191.468545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761191.468569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761191.468575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761191.468586:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1195 00000100:00080000:1.0:1789761191.468598:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761191.468606:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761191.468661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493921408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761191.468688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761191.468691:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761191.468692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761191.468696:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1195 00000100:00080000:1.0:1789761191.468698:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761191.468701:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761191.468711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761191.468721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493921536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761191.468731:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493921408:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761191.468732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761191.468741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761191.468743:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761191.468747:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1195 00000100:00080000:1.0:1789761191.468749:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761191.468751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761191.468754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761191.468764:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493921664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761191.468771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761191.468773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761191.468775:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761191.468779:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1195 00000100:00080000:1.0:1789761191.468795:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761191.468801:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761191.468806:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761191.468811:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761191.468822:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493921792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761191.468858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761191/real 1789761191] req@ffff8eb65016c380 x1876699493921408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761191.468874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493921408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761191.468884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761191.468888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761191.468891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493921408:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761191.468913:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761191.468929:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761191.468938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761191/real 1789761191] req@ffff8eb65016d500 x1876699493921536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761191.468946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493921536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761191.468952:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761191.468955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761191.468957:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921536:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761191.468962:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921664:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761191.468992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761191.469000:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761191/real 1789761191] req@ffff8eb65016fb80 x1876699493921664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761191.469011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493921664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761191.469020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761191.469024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761191.469028:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493921664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761191.469035:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493921792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761191.469056:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761191.469068:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761191/real 1789761191] req@ffff8eb65016e300 x1876699493921792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761191.469080:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493921792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761191.469086:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761191.469088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761191.469091:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493921792:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761192.172607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761193.196580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761194.220579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761195.244633:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761196.268588:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761196.588526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761196.588541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761196.588546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761196.588557:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1201 00000100:00080000:0.0:1789761196.588569:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761196.588576:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761196.588617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493921920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761196.588642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761196.588646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761196.588647:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761196.588650:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1201 00000100:00080000:0.0:1789761196.588654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761196.588659:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761196.588671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761196.588683:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493921920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761196.588688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493922048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761196.588711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761196.588714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761196.588716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761196.588719:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1201 00000100:00080000:0.0:1789761196.588722:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761196.588725:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761196.588730:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761196.588741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493922176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761196.588749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761196.588751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761196.588752:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761196.588755:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1201 00000100:00080000:0.0:1789761196.588758:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761196.588761:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761196.588763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761196.588771:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493922304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761196.588780:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761196.588829:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761196/real 1789761196] req@ffff8eb64304c000 x1876699493921920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761196.588850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493921920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761196.588864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761196.588869:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761196.588874:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493921920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761196.588903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761196.588923:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761196.588948:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761196/real 1789761196] req@ffff8eb64304d180 x1876699493922048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761196.588957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493922048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761196.588968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761196.588970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761196.588973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922048:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761196.588977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922176:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761196.588993:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761196.588999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761196/real 1789761196] req@ffff8eb64304ed80 x1876699493922176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761196.589007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493922176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761196.589013:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761196.589015:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761196.589017:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922176:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761196.589020:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493922304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761196.589031:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761196.589041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761196/real 1789761196] req@ffff8eb64304e300 x1876699493922304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761196.589049:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493922304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761196.589054:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761196.589056:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761196.589059:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493922304:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761197.292599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761198.316513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761199.340580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761200.364528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761201.388554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761201.708470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761201.708482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761201.708487:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761201.708497:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1206 00000100:00080000:0.0:1789761201.708508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761201.708514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761201.708547:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493922432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761201.708577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761201.708581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761201.708583:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761201.708587:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1206 00000100:00080000:0.0:1789761201.708592:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761201.708597:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761201.708606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761201.708620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493922560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761201.708629:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493922432:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761201.708631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761201.708634:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761201.708636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761201.708640:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1206 00000100:00080000:0.0:1789761201.708643:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761201.708646:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761201.708651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761201.708660:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493922688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761201.708669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761201.708672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761201.708673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761201.708677:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1206 00000100:00080000:0.0:1789761201.708679:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761201.708683:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789761201.708686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789761201.708688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761201.708696:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493922816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761201.708723:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761201/real 1789761201] req@ffff8eb64304e300 x1876699493922432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761201.708742:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493922432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761201.708752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761201.708754:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761201.708757:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493922432:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761201.708777:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761201.708795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761201.708804:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761201/real 1789761201] req@ffff8eb64304ed80 x1876699493922560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761201.708812:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493922560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761201.708817:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761201.708820:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761201.708822:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922560:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761201.708825:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922688:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761201.708837:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761201.708845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761201/real 1789761201] req@ffff8eb64304d180 x1876699493922688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761201.708857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493922688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761201.708865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761201.708868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761201.708872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493922688:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761201.708878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493922816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761201.708915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761201.708925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761201/real 1789761201] req@ffff8eb64304c000 x1876699493922816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761201.708933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493922816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761201.708939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761201.708941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761201.708943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493922816:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761202.412521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761203.436525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761204.460505:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761205.484537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761206.508513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761206.828469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761206.828478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761206.828482:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761206.828491:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1211 00000100:00080000:2.0:1789761206.828500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761206.828506:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761206.828538:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493922944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761206.828566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761206.828569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761206.828570:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761206.828573:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1211 00000100:00080000:2.0:1789761206.828582:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761206.828585:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761206.828592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761206.828602:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493923072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761206.828612:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493922944:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761206.828612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761206.828622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761206.828624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761206.828628:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1211 00000100:00080000:2.0:1789761206.828633:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761206.828638:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761206.828644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761206.828659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493923200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761206.828670:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761206.828672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761206.828674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761206.828677:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1211 00000100:00080000:2.0:1789761206.828680:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761206.828684:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761206.828688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761206.828697:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493923328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761206.828705:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761206.828736:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761206.828754:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493923328:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761207.532507:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761208.556536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761209.580565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761209.900614:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761209.900656:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761209.900673:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761209.900713:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761209.900727:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761209.900736:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789761209.900761:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761206/real 1789761209] req@ffff8eb650f81c00 x1876699493923328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.900789:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493923328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.900801:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.900807:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761209.900811:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493923328:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761209.900820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761206/real 1789761209] req@ffff8eb650f82680 x1876699493923072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.900851:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493923072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.900861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.900865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761209.900869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761209.900875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761206/real 1789761209] req@ffff8eb650f82d80 x1876699493923200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.900888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493923200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.900898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.900901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761209.900905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761209.900912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761206/real 1789761209] req@ffff8eb650f81880 x1876699493922944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761209.900923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493922944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761209.900930:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761209.900932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761209.900936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493922944:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789761210.604517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761211.628531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761211.948507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761211.948519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761211.948524:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761211.948535:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1216 00000100:00080000:1.0:1789761211.948546:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761211.948552:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761211.948582:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493923456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761211.948608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761211.948611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761211.948613:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761211.948616:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1216 00000100:00080000:1.0:1789761211.948619:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761211.948622:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761211.948630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761211.948639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493923584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761211.948659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761211.948662:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493923456:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761211.948672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761211.948674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761211.948679:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1216 00000100:00080000:1.0:1789761211.948684:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761211.948689:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761211.948697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761211.948712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493923712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761211.948722:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761211.948724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761211.948729:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761211.948732:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761211.948737:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1216 00000100:00080000:1.0:1789761211.948742:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761211.948746:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761211.948752:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761211.948766:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493923840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761211.948773:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761211/real 1789761211] req@ffff8eb65016dc00 x1876699493923456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761211.948801:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493923456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761211.948824:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761211.948829:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761211.948833:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493923456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761211.948863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761211.948889:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761211.948903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761211/real 1789761211] req@ffff8eb65016ce00 x1876699493923584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761211.948915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493923584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761211.948924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761211.948927:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761211.948931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923584:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761211.948939:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923712:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761211.948962:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761211.948974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761211/real 1789761211] req@ffff8eb65016c700 x1876699493923712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761211.948984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493923712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761211.948991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761211.948994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761211.948997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493923712:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761211.949002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493923840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761211.949021:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761211.949031:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761211/real 1789761211] req@ffff8eb65016df80 x1876699493923840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761211.949043:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493923840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761211.949050:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761211.949053:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761211.949056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493923840:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761212.652545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761213.676528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761214.700580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761215.724568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761216.748581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761217.068478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761217.068488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761217.068492:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761217.068507:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1221 00000100:00080000:0.0:1789761217.068517:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761217.068524:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761217.068559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493923968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761217.068580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761217.068583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761217.068584:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761217.068587:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1221 00000100:00080000:0.0:1789761217.068590:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761217.068594:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761217.068602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761217.068615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493924096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761217.068615:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493923968:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761217.068629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761217.068632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761217.068633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761217.068635:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1221 00000100:00080000:0.0:1789761217.068637:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761217.068640:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761217.068644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761217.068652:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493924224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761217.068660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761217.068663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761217.068665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761217.068667:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1221 00000100:00080000:0.0:1789761217.068669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761217.068672:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789761217.068676:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789761217.068679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761217.068689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493924352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761217.068711:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761217/real 1789761217] req@ffff8eb64304f800 x1876699493923968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.068726:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493923968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.068739:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.068742:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761217.068745:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493923968:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761217.068764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761217.068795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761217.068805:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761217/real 1789761217] req@ffff8eb64304c380 x1876699493924096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.068813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493924096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.068819:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.068821:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761217.068824:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924096:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761217.068830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924224:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761217.068852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761217.068864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761217/real 1789761217] req@ffff8eb64304dc00 x1876699493924224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.068878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493924224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.068886:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.068889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761217.068891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924224:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761217.068896:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493924352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761217.068911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761217.068919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761217/real 1789761217] req@ffff8eb64304ce00 x1876699493924352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761217.068926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493924352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761217.068931:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761217.068933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761217.068935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493924352:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761217.772528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761218.796562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761219.820575:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761220.844602:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761221.868521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761222.188511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761222.188520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761222.188523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761222.188531:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1226 00000100:00080000:0.0:1789761222.188540:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761222.188545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761222.188578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493924480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761222.188601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761222.188603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761222.188604:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761222.188607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1226 00000100:00080000:0.0:1789761222.188610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761222.188612:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761222.188620:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761222.188627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493924608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761222.188635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761222.188637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761222.188639:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761222.188641:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1226 00000100:00080000:0.0:1789761222.188643:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761222.188645:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761222.188648:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761222.188658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493924736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761222.188659:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493924480:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761222.188671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761222.188673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761222.188674:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761222.188676:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1226 00000100:00080000:0.0:1789761222.188678:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761222.188680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761222.188683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761222.188691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493924864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761222.188755:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761222.188795:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761222/real 1789761222] req@ffff8eb64304d500 x1876699493924480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761222.188813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493924480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761222.188823:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761222.188828:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761222.188832:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493924480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761222.188839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761222.188861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761222.188872:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761222/real 1789761222] req@ffff8eb64304d880 x1876699493924608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761222.188885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493924608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761222.188893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761222.188896:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761222.188900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761222.188925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924736:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761222.188945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761222.188955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761222/real 1789761222] req@ffff8eb64304f480 x1876699493924736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761222.188967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493924736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761222.188980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761222.188984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761222.188987:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493924736:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761222.188993:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493924864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761222.189008:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761222.189018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761222/real 1789761222] req@ffff8eb64304e680 x1876699493924864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761222.189028:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493924864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761222.189036:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761222.189039:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761222.189042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493924864:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761222.892488:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761223.916536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761224.940575:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761225.964507:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761226.988554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761227.308494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761227.308505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761227.308510:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761227.308520:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1231 00000100:00080000:1.0:1789761227.308532:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761227.308538:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761227.308570:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493924992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761227.308599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761227.308601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761227.308603:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761227.308606:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1231 00000100:00080000:1.0:1789761227.308608:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761227.308611:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761227.308618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761227.308628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493925120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761227.308637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761227.308638:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493924992:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761227.308642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761227.308646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761227.308650:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1231 00000100:00080000:1.0:1789761227.308653:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761227.308655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761227.308658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761227.308668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493925248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761227.308676:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761227.308678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761227.308680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761227.308682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1231 00000100:00080000:1.0:1789761227.308694:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761227.308714:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761227.308718:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761227.308724:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761227.308738:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493925376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761227.308764:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761227/real 1789761227] req@ffff8eb65016c000 x1876699493924992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761227.308777:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493924992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761227.308785:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761227.308788:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761227.308791:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493924992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761227.308808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761227.308824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761227.308832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761227/real 1789761227] req@ffff8eb65016f480 x1876699493925120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761227.308840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493925120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761227.308845:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761227.308847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761227.308849:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925120:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761227.308853:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925248:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761227.308865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761227.308871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761227/real 1789761227] req@ffff8eb65016ed80 x1876699493925248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761227.308878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493925248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761227.308883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761227.308885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761227.308887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925248:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761227.308890:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493925376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761227.308899:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761227.308905:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761227/real 1789761227] req@ffff8eb65016d180 x1876699493925376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761227.308914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493925376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761227.308922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761227.308925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761227.308928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493925376:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761228.012550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761229.036540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761230.060514:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761231.084541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761232.108513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761232.428520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761232.428527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761232.428531:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761232.428539:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1236 00000100:00080000:1.0:1789761232.428547:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761232.428553:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761232.428588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493925504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761232.428619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761232.428623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761232.428625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761232.428630:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1236 00000100:00080000:1.0:1789761232.428634:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761232.428638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761232.428648:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761232.428702:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493925632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761232.428706:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493925504:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761232.428723:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761232.428726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761232.428728:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761232.428731:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1236 00000100:00080000:1.0:1789761232.428735:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761232.428739:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761232.428743:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761232.428755:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493925760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761232.428763:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761232.428766:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761232.428768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761232.428770:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761232.428773:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1236 00000100:00080000:1.0:1789761232.428777:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761232.428780:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761232.428784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761232.428792:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493925888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761232.428806:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761232/real 1789761232] req@ffff8eb65016f100 x1876699493925504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761232.428821:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493925504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761232.428830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761232.428833:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761232.428836:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493925504:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761232.428854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761232.428868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761232.428879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761232/real 1789761232] req@ffff8eb65016ca80 x1876699493925632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761232.428892:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493925632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761232.428901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761232.428905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761232.428908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761232.428914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925760:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761232.428933:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761232.428944:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761232/real 1789761232] req@ffff8eb65016d180 x1876699493925760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761232.428960:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493925760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761232.428970:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761232.428972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761232.428975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493925760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761232.428979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493925888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761232.428992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761232.428999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761232/real 1789761232] req@ffff8eb65016ed80 x1876699493925888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761232.429011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493925888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761232.429016:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761232.429018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761232.429020:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493925888:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761233.132522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761234.156523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761235.180560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761236.204557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761237.228545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761237.548529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761237.548537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761237.548540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761237.548548:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1242 00000100:00080000:2.0:1789761237.548557:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761237.548563:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761237.548598:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493926016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761237.548638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761237.548641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761237.548643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761237.548646:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1242 00000100:00080000:2.0:1789761237.548650:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761237.548653:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761237.548664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761237.548680:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493926144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761237.548687:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493926016:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761237.548699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761237.548703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761237.548706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761237.548710:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1242 00000100:00080000:2.0:1789761237.548714:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761237.548717:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761237.548722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761237.548732:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493926272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761237.548740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761237.548743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761237.548744:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761237.548748:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761237.548752:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1242 00000100:00080000:2.0:1789761237.548756:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761237.548759:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761237.548763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761237.548771:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493926400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761237.548791:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761237/real 1789761237] req@ffff8eb650f81880 x1876699493926016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761237.548806:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493926016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761237.548814:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761237.548817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761237.548820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493926016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761237.548838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761237.548852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761237.548859:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761237/real 1789761237] req@ffff8eb650f82d80 x1876699493926144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761237.548867:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493926144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761237.548872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761237.548874:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761237.548876:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926144:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761237.548880:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926272:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761237.548890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761237.548896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761237/real 1789761237] req@ffff8eb650f82680 x1876699493926272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761237.548904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493926272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761237.548909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761237.548910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761237.548912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926272:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761237.548916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493926400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761237.548926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761237.548931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761237/real 1789761237] req@ffff8eb650f81c00 x1876699493926400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761237.548939:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493926400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761237.548943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761237.548945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761237.548947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493926400:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761238.252567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761239.276563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761240.300563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761241.324533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761242.348615:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761242.668479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761242.668487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761242.668491:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761242.668499:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1247 00000100:00080000:1.0:1789761242.668508:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761242.668513:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761242.668543:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493926528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761242.668568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761242.668572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761242.668574:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761242.668579:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1247 00000100:00080000:1.0:1789761242.668595:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761242.668601:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761242.668612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761242.668629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493926656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761242.668643:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493926528:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761242.668646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761242.668658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761242.668661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761242.668665:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1247 00000100:00080000:1.0:1789761242.668668:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761242.668671:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761242.668676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761242.668686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493926784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761242.668698:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761242.668701:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761242.668702:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761242.668705:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1247 00000100:00080000:1.0:1789761242.668727:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761242.668729:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761242.668732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761242.668739:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493926912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761242.668749:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761242.668807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761242/real 1789761242] req@ffff8eb65016f480 x1876699493926528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761242.668823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493926528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761242.668834:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761242.668837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761242.668840:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493926528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761242.668865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761242.668883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761242.668892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761242/real 1789761242] req@ffff8eb65016c000 x1876699493926656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761242.668905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493926656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761242.668910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761242.668912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761242.668914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761242.668919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926784:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761242.668931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761242.668938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761242/real 1789761242] req@ffff8eb65016df80 x1876699493926784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761242.668946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493926784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761242.668951:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761242.668953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761242.668955:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493926784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761242.668959:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493926912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761242.668969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761242.668975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761242/real 1789761242] req@ffff8eb65016c700 x1876699493926912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761242.668982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493926912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761242.668992:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761242.668994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761242.668997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493926912:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761243.372538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761244.396624:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761245.420535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761246.444683:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761247.468603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761247.788485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761247.788496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761247.788500:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761247.788511:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1252 00000100:00080000:1.0:1789761247.788524:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761247.788531:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761247.788596:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493927040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761247.788625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761247.788629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761247.788631:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761247.788636:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1252 00000100:00080000:1.0:1789761247.788641:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761247.788646:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761247.788658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761247.788671:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493927040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761247.788672:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493927168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761247.788683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761247.788687:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761247.788688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761247.788692:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1252 00000100:00080000:1.0:1789761247.788695:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761247.788699:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761247.788703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761247.788713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493927296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761247.788722:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761247.788725:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761247.788727:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761247.788730:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1252 00000100:00080000:1.0:1789761247.788733:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761247.788737:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761247.788741:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761247.788749:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761247.788750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493927424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761247.788800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761247/real 1789761247] req@ffff8eb65016ce00 x1876699493927040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761247.788817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493927040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761247.788826:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761247.788829:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761247.788832:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493927040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761247.788854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761247.788873:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761247.788882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761247/real 1789761247] req@ffff8eb65016dc00 x1876699493927168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761247.788889:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493927168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761247.788895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761247.788897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761247.788899:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761247.788903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927296:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761247.788920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761247.788927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761247/real 1789761247] req@ffff8eb65016e300 x1876699493927296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761247.788934:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493927296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761247.788940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761247.788941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761247.788943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927296:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761247.788947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493927424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761247.788959:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761247.788965:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761247/real 1789761247] req@ffff8eb65016fb80 x1876699493927424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761247.788972:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493927424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761247.788977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761247.788978:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761247.788980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493927424:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761248.492611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761249.516586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761250.540594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761251.564597:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761252.588590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761252.908494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761252.908502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761252.908517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761252.908528:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1257 00000100:00080000:1.0:1789761252.908537:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761252.908543:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761252.908573:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493927552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761252.908596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761252.908598:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761252.908600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761252.908602:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1257 00000100:00080000:1.0:1789761252.908605:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761252.908607:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761252.908615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761252.908622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493927680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761252.908629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761252.908632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761252.908634:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761252.908639:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1257 00000100:00080000:1.0:1789761252.908642:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761252.908646:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761252.908652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761252.908667:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493927808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761252.908682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493927552:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761252.908685:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761252.908701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761252.908703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761252.908711:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1257 00000100:00080000:1.0:1789761252.908715:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761252.908720:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761252.908726:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761252.908743:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493927936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761252.908782:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761252.908832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761252/real 1789761252] req@ffff8eb65016d500 x1876699493927552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761252.908849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493927552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761252.908858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761252.908861:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761252.908865:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493927552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761252.908871:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761252.908888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761252.908896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761252/real 1789761252] req@ffff8eb65016c380 x1876699493927680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761252.908905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493927680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761252.908910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761252.908912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761252.908914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927680:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761252.908938:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927808:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761252.908951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761252.908958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761252/real 1789761252] req@ffff8eb65016f800 x1876699493927808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761252.908966:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493927808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761252.908971:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761252.908973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761252.908975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493927808:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761252.908979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493927936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761252.908990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761252.908996:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761252/real 1789761252] req@ffff8eb65016e680 x1876699493927936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761252.909010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493927936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761252.909015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761252.909018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761252.909020:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493927936:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761253.612590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761254.636617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761255.660552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761256.684596:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761257.708567:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761258.028503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761258.028518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761258.028522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761258.028531:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1262 00000100:00080000:0.0:1789761258.028540:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761258.028546:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761258.028578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493928064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761258.028600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761258.028603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761258.028604:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761258.028607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1262 00000100:00080000:0.0:1789761258.028610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761258.028612:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761258.028619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761258.028628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493928192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761258.028636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761258.028639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761258.028643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761258.028643:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493928064:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761258.028652:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1262 00000100:00080000:0.0:1789761258.028657:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761258.028661:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761258.028667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761258.028680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493928320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761258.028692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761258.028704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761258.028707:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761258.028714:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1262 00000800:00000100:1.0:1789761258.028715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789761258.028722:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761258.028729:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761258.028735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761258.028749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493928448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761258.028762:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761258/real 1789761258] req@ffff8eb64304ca80 x1876699493928064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761258.028783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493928064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761258.028804:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761258.028809:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761258.028813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493928064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761258.028837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761258.028862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761258.028875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761258/real 1789761258] req@ffff8eb64304ea00 x1876699493928192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761258.028890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493928192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761258.028900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761258.028904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761258.028907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761258.028914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928320:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761258.028934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761258.028944:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761258/real 1789761258] req@ffff8eb64304c700 x1876699493928320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761258.028958:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493928320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761258.028968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761258.028971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761258.028982:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761258.028990:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493928448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761258.029010:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761258.029022:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761258/real 1789761258] req@ffff8eb64304f100 x1876699493928448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761258.029035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493928448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761258.029045:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761258.029048:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761258.029052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493928448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761258.732559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761259.756562:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761260.780519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761261.804550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761262.828528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761263.148540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761263.148549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761263.148553:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761263.148569:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1267 00000100:00080000:1.0:1789761263.148578:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761263.148583:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761263.148618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493928576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761263.148639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761263.148642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761263.148643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761263.148646:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1267 00000100:00080000:1.0:1789761263.148649:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761263.148651:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761263.148660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761263.148669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493928704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761263.148675:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493928576:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761263.148680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761263.148685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761263.148687:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761263.148690:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1267 00000100:00080000:1.0:1789761263.148692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761263.148694:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761263.148697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761263.148706:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493928832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761263.148713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761263.148715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761263.148716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761263.148719:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1267 00000100:00080000:1.0:1789761263.148730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761263.148735:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761263.148741:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761263.148741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761263.148758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493928960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761263.148775:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761263/real 1789761263] req@ffff8eb65016ea00 x1876699493928576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761263.148790:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493928576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761263.148799:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761263.148801:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761263.148805:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493928576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761263.148824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761263.148839:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761263.148847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761263/real 1789761263] req@ffff8eb65016d880 x1876699493928704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761263.148855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493928704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761263.148865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761263.148867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761263.148870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928704:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761263.148873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928832:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761263.148884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761263.148890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761263/real 1789761263] req@ffff8eb65016e680 x1876699493928832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761263.148897:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493928832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761263.148902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761263.148904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761263.148906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493928832:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761263.148910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493928960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761263.148920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761263.148925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761263/real 1789761263] req@ffff8eb65016f800 x1876699493928960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761263.148932:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493928960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761263.148937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761263.148939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761263.148941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493928960:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789761263.852557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761264.876565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761265.900527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761266.924541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761267.948553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761268.268499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761268.268514:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761268.268520:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761268.268533:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1272 00000100:00080000:1.0:1789761268.268547:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761268.268557:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761268.268606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493929088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761268.268634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761268.268637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761268.268639:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761268.268642:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1272 00000100:00080000:1.0:1789761268.268645:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761268.268648:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761268.268658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761268.268675:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493929216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761268.268684:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493929088:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761268.268686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761268.268692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761268.268696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761268.268700:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1272 00000100:00080000:1.0:1789761268.268704:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761268.268708:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761268.268714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761268.268727:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493929344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761268.268751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761268.268756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761268.268758:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761268.268763:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1272 00000100:00080000:1.0:1789761268.268768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761268.268772:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761268.268778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761268.268779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761268.268795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493929472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761268.268827:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761268/real 1789761268] req@ffff8eb65016c380 x1876699493929088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761268.268844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493929088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761268.268853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761268.268856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761268.268859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493929088:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761268.268883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761268.268901:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761268.268909:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761268/real 1789761268] req@ffff8eb65016d500 x1876699493929216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761268.268918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493929216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761268.268924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761268.268926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761268.268929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929216:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761268.268933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929344:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761268.268953:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761268.268966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761268/real 1789761268] req@ffff8eb65016fb80 x1876699493929344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761268.268980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493929344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761268.268989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761268.268993:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761268.268996:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929344:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761268.269003:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493929472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761268.269027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761268.269039:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761268/real 1789761268] req@ffff8eb65016e300 x1876699493929472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761268.269052:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493929472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761268.269061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761268.269065:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761268.269069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493929472:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761268.972513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761269.996544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761271.020571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761272.044532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761273.068519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761273.388457:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761273.388465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761273.388468:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761273.388476:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1277 00000100:00080000:1.0:1789761273.388485:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761273.388491:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761273.388521:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493929600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761273.388542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761273.388545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761273.388546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761273.388549:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1277 00000100:00080000:1.0:1789761273.388551:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761273.388554:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761273.388568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761273.388577:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493929728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761273.388582:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493929600:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761273.388588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761273.388595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761273.388596:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761273.388599:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1277 00000100:00080000:1.0:1789761273.388601:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761273.388604:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761273.388607:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761273.388616:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493929856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761273.388623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761273.388626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761273.388627:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761273.388630:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1277 00000100:00080000:1.0:1789761273.388635:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761273.388638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761273.388640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761273.388648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493929984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761273.388694:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761273.388728:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761273.388743:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493929984:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761274.092537:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761275.116549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761276.140556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761276.460686:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761276.460735:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761276.460749:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761276.460793:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761276.460799:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761276.460805:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761276.460845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761273/real 1789761276] req@ffff8eb65016ce00 x1876699493929728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.460872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493929728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.460885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.460892:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761276.460896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761276.460905:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761273/real 1789761276] req@ffff8eb65016c700 x1876699493929856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.460920:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493929856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.460930:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.460933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761276.460936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493929856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761276.460943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761273/real 1789761276] req@ffff8eb65016df80 x1876699493929984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.460956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493929984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.460964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.460968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761276.460972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493929984:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761276.460979:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761273/real 1789761276] req@ffff8eb65016dc00 x1876699493929600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.460992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493929600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.461000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.461003:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761276.461007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493929600:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761277.164583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761278.188522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761278.508487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761278.508498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761278.508502:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761278.508511:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1283 00000100:00080000:1.0:1789761278.508522:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761278.508530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761278.508560:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493930112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761278.508582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761278.508585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761278.508586:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761278.508589:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1283 00000100:00080000:1.0:1789761278.508592:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761278.508594:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761278.508602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761278.508610:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493930240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761278.508617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761278.508620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761278.508621:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761278.508624:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1283 00000100:00080000:1.0:1789761278.508625:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761278.508628:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761278.508631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761278.508639:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493930112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761278.508640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493930368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761278.508652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761278.508654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761278.508655:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761278.508658:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1283 00000100:00080000:1.0:1789761278.508660:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761278.508663:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761278.508668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761278.508681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493930496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761278.508698:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761278.508731:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761278/real 1789761278] req@ffff8eb65016c000 x1876699493930112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761278.508748:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493930112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761278.508759:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761278.508764:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761278.508768:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493930112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761278.508776:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761278.508799:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761278.508812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761278/real 1789761278] req@ffff8eb65016f480 x1876699493930240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761278.508826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493930240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761278.508836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761278.508840:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761278.508844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930240:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761278.508870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930368:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761278.508893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761278.508906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761278/real 1789761278] req@ffff8eb65016ed80 x1876699493930368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761278.508918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493930368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761278.508926:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761278.508930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761278.508933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930368:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761278.508939:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493930496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761278.508961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761278.508979:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761278/real 1789761278] req@ffff8eb65016d180 x1876699493930496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761278.508990:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493930496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761278.509003:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761278.509006:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761278.509010:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493930496:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761279.212544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761280.236516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761281.260565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761282.284506:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761283.308571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761283.628454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761283.628462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761283.628466:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761283.628474:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1288 00000100:00080000:1.0:1789761283.628482:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761283.628488:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761283.628520:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493930624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761283.628542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761283.628545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761283.628546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761283.628549:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1288 00000100:00080000:1.0:1789761283.628552:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761283.628554:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761283.628562:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761283.628569:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493930752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761283.628577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761283.628579:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761283.628581:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761283.628583:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1288 00000100:00080000:1.0:1789761283.628586:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761283.628589:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761283.628593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761283.628601:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493930880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761283.628609:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761283.628611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761283.628612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761283.628615:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1288 00000100:00080000:1.0:1789761283.628617:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761283.628619:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761283.628622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761283.628631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493931008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761283.628634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493930624:192.168.203.160@tcp:250:kworker.0 00000800:00000100:0.0:1789761283.628723:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761283.628770:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761283/real 1789761283] req@ffff8eb65016ca80 x1876699493930624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761283.628785:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493930624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761283.628794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761283.628798:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761283.628801:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493930624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761283.628808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761283.628823:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761283.628832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761283/real 1789761283] req@ffff8eb65016f100 x1876699493930752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761283.628840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493930752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761283.628845:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761283.628848:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761283.628850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761283.628855:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930880:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761283.628866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761283.628873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761283/real 1789761283] req@ffff8eb65016d180 x1876699493930880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761283.628879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493930880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761283.628885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761283.628887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761283.628889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493930880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761283.628915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493931008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761283.628927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761283.628933:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761283/real 1789761283] req@ffff8eb65016ed80 x1876699493931008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761283.628941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493931008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761283.628946:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761283.628948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761283.628950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493931008:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761284.332572:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761285.356515:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761286.380505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761287.404561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761288.428594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761288.748513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761288.748524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761288.748536:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761288.748548:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1293 00000100:00080000:1.0:1789761288.748578:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761288.748585:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761288.748634:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493931136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761288.748669:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761288.748672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761288.748674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761288.748678:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1293 00000100:00080000:1.0:1789761288.748681:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761288.748684:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761288.748695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761288.748719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493931264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761288.748734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761288.748739:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493931136:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761288.748741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761288.748747:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761288.748754:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1293 00000100:00080000:1.0:1789761288.748760:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761288.748766:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761288.748772:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761288.748787:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493931392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761288.748802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761288.748805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761288.748807:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761288.748811:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1293 00000100:00080000:1.0:1789761288.748814:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761288.748817:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761288.748821:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761288.748822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761288.748834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493931520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761288.748873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761288/real 1789761288] req@ffff8eb65016f480 x1876699493931136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761288.748890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493931136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761288.748901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761288.748904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761288.748908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493931136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761288.748932:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761288.748950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761288.748959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761288/real 1789761288] req@ffff8eb65016c000 x1876699493931264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761288.748969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493931264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761288.748975:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761288.748978:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761288.748980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761288.748984:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931392:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761288.748997:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761288.749003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761288/real 1789761288] req@ffff8eb65016dc00 x1876699493931392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761288.749017:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493931392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761288.749023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761288.749025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761288.749027:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931392:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761288.749031:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493931520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761288.749041:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761288.749048:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761288/real 1789761288] req@ffff8eb65016df80 x1876699493931520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761288.749055:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493931520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761288.749061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761288.749063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761288.749065:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493931520:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761289.452548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761290.476537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761291.500549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761292.524497:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761293.548541:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761293.868494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761293.868502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761293.868505:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761293.868513:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1298 00000100:00080000:2.0:1789761293.868529:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761293.868538:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761293.868570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493931648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761293.868592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761293.868594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761293.868596:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761293.868599:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1298 00000100:00080000:2.0:1789761293.868601:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761293.868603:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761293.868611:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761293.868619:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493931776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761293.868627:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761293.868631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761293.868633:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493931648:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761293.868634:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761293.868641:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1298 00000100:00080000:2.0:1789761293.868644:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761293.868646:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761293.868650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761293.868658:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493931904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761293.868666:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761293.868668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761293.868669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761293.868672:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1298 00000100:00080000:2.0:1789761293.868675:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761293.868683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761293.868686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761293.868694:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761293.868697:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493932032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761293.868732:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761293/real 1789761293] req@ffff8eb650f82a00 x1876699493931648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761293.868746:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493931648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761293.868754:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761293.868757:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761293.868760:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493931648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761293.868780:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761293.868804:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761293.868817:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761293/real 1789761293] req@ffff8eb650f80700 x1876699493931776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761293.868830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493931776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761293.868836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761293.868838:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761293.868842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931776:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761293.868848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931904:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761293.868871:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761293.868883:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761293/real 1789761293] req@ffff8eb650f80e00 x1876699493931904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761293.868895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493931904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761293.868901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761293.868903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761293.868905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493931904:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761293.868910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493932032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761293.868924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761293.868932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761293/real 1789761293] req@ffff8eb650f83480 x1876699493932032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761293.868945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493932032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761293.868953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761293.868956:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761293.868960:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493932032:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761294.572547:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761295.596654:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761296.620543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761297.644516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761298.668554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761298.988493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761298.988504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761298.988509:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761298.988519:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1303 00000100:00080000:1.0:1789761298.988531:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761298.988538:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761298.988577:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493932160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761298.988604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761298.988608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761298.988610:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761298.988615:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1303 00000100:00080000:1.0:1789761298.988620:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761298.988625:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761298.988636:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493932160:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789761298.988637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761298.988655:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493932288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761298.988667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761298.988672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761298.988674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761298.988678:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1303 00000100:00080000:1.0:1789761298.988683:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761298.988688:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761298.988694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761298.988696:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761298.988710:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493932416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761298.988724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761298.988728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761298.988730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761298.988735:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1303 00000100:00080000:1.0:1789761298.988740:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761298.988741:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761298/real 1789761298] req@ffff8eb65016c700 x1876699493932160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761298.988748:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761298.988760:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493932160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761298.988765:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761298.988771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.988775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761298.988779:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493932160:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761298.988783:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493932544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761298.988802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761298.988824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761298.988833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761298/real 1789761298] req@ffff8eb65016ce00 x1876699493932288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.988841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493932288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.988847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.988849:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761298.988851:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761298.988855:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932416:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761298.988865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761298.988871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761298/real 1789761298] req@ffff8eb65016e300 x1876699493932416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.988878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493932416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.988883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.988885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761298.988896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761298.988899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493932544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761298.988911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761298.988918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761298/real 1789761298] req@ffff8eb65016fb80 x1876699493932544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761298.988931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493932544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761298.988935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761298.988937:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761298.988939:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493932544:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761299.692532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761300.716532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761301.740532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761302.764617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761303.788506:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761304.108555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761304.108567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761304.108572:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761304.108583:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1308 00000100:00080000:1.0:1789761304.108595:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761304.108603:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761304.108639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493932672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761304.108671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761304.108675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761304.108677:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761304.108682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1308 00000100:00080000:1.0:1789761304.108686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761304.108692:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761304.108718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761304.108735:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493932672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761304.108736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493932800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761304.108762:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761304.108767:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761304.108769:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761304.108773:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1308 00000100:00080000:1.0:1789761304.108791:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761304.108796:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761304.108803:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761304.108804:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761304.108821:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493932928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761304.108840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761304.108844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761304.108846:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761304.108850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761304/real 1789761304] req@ffff8eb65016d500 x1876699493932672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761304.108854:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1308 00000100:00080000:1.0:1789761304.108861:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761304.108868:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761304.108875:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761304.108889:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493932672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761304.108890:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493933056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761304.108925:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761304.108930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761304.108934:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493932672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761304.108960:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761304.108982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761304.108991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761304/real 1789761304] req@ffff8eb65016c380 x1876699493932800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761304.108999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493932800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761304.109010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761304.109012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761304.109014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932800:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761304.109018:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932928:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761304.109030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761304.109036:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761304/real 1789761304] req@ffff8eb65016f800 x1876699493932928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761304.109043:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493932928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761304.109048:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761304.109050:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761304.109052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493932928:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761304.109055:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493933056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761304.109065:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761304.109071:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761304/real 1789761304] req@ffff8eb65016e680 x1876699493933056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761304.109078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493933056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761304.109083:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761304.109084:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761304.109087:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493933056:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761304.812574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761305.836520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761306.860595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761307.884556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761308.908537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761309.228457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761309.228465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761309.228469:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761309.228478:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1313 00000100:00080000:1.0:1789761309.228487:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761309.228493:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761309.228524:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493933184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761309.228546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761309.228549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761309.228550:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761309.228553:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1313 00000100:00080000:1.0:1789761309.228556:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761309.228558:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761309.228566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761309.228575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493933312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761309.228582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761309.228585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761309.228586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761309.228588:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1313 00000100:00080000:1.0:1789761309.228590:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761309.228595:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493933184:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761309.228596:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761309.228606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761309.228621:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493933440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761309.228633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761309.228637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761309.228639:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761309.228644:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1313 00000100:00080000:1.0:1789761309.228648:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761309.228653:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761309.228655:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761309.228663:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761309.228677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493933568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761309.228689:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761309/real 1789761309] req@ffff8eb65016d880 x1876699493933184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761309.228703:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493933184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761309.228710:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761309.228714:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761309.228717:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493933184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761309.228722:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761309.228736:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761309.228748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761309/real 1789761309] req@ffff8eb65016ea00 x1876699493933312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761309.228761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493933312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761309.228770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761309.228773:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761309.228776:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933312:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761309.228798:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933440:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761309.228814:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761309.228822:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761309/real 1789761309] req@ffff8eb65016e680 x1876699493933440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761309.228829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493933440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761309.228835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761309.228837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761309.228839:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761309.228842:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493933568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761309.228852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761309.228857:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761309/real 1789761309] req@ffff8eb65016f800 x1876699493933568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761309.228864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493933568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761309.228870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761309.228873:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761309.228876:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493933568:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761309.932517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761310.956544:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761311.980523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761313.004540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761314.028524:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761314.348492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761314.348501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761314.348505:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761314.348512:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1318 00000100:00080000:2.0:1789761314.348521:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761314.348527:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761314.348561:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493933696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761314.348580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761314.348583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761314.348584:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761314.348588:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1318 00000100:00080000:2.0:1789761314.348590:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761314.348592:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761314.348600:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761314.348609:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493933824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761314.348617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761314.348620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761314.348621:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761314.348623:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1318 00000100:00080000:2.0:1789761314.348625:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761314.348628:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761314.348633:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493933696:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761314.348633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761314.348642:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493933952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761314.348650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761314.348652:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761314.348653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761314.348656:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1318 00000100:00080000:2.0:1789761314.348658:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761314.348661:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761314.348664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761314.348674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493934080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761314.348695:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761314.348728:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761314/real 1789761314] req@ffff8eb650f83b80 x1876699493933696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761314.348741:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493933696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761314.348749:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761314.348752:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761314.348755:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493933696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761314.348760:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761314.348773:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761314.348781:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761314/real 1789761314] req@ffff8eb650f81f80 x1876699493933824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761314.348789:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493933824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761314.348794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761314.348796:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761314.348798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761314.348817:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933952:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761314.348827:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761314.348834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761314/real 1789761314] req@ffff8eb650f81180 x1876699493933952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761314.348841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493933952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761314.348851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761314.348853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761314.348855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493933952:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761314.348858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493934080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761314.348867:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761314.348873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761314/real 1789761314] req@ffff8eb650f83800 x1876699493934080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761314.348879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493934080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761314.348884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761314.348886:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761314.348888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493934080:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761315.052526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761316.076571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761317.100545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761318.124500:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761319.148555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761319.468520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761319.468531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761319.468535:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761319.468546:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1323 00000100:00080000:2.0:1789761319.468556:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761319.468564:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761319.468600:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493934208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761319.468633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761319.468637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761319.468639:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761319.468643:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1323 00000100:00080000:2.0:1789761319.468647:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761319.468651:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761319.468660:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761319.468675:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493934336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761319.468682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493934208:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761319.468690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761319.468696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761319.468697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761319.468703:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1323 00000100:00080000:2.0:1789761319.468707:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761319.468711:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761319.468717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761319.468731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493934464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761319.468741:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761319.468745:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761319.468747:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761319.468750:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1323 00000100:00080000:2.0:1789761319.468755:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761319.468759:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761319.468764:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761319.468771:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761319.468784:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493934592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761319.468801:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761319/real 1789761319] req@ffff8eb650f80a80 x1876699493934208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761319.468817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493934208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761319.468826:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761319.468829:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761319.468832:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493934208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761319.468849:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761319.468867:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761319.468875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761319/real 1789761319] req@ffff8eb650f80000 x1876699493934336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761319.468883:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493934336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761319.468889:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761319.468891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761319.468893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934336:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761319.468897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934464:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789761319.468907:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761319.468913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761319/real 1789761319] req@ffff8eb650f80380 x1876699493934464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761319.468925:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493934464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761319.468930:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761319.468932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761319.468934:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934464:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789761319.468937:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493934592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789761319.468947:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761319.468953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761319/real 1789761319] req@ffff8eb650f81500 x1876699493934592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761319.468960:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493934592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761319.468965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761319.468966:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789761319.468968:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493934592:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761320.172579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761321.196527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761322.220568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761323.244498:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761324.268595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761324.588490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761324.588501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761324.588505:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761324.588516:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1329 00000100:00080000:1.0:1789761324.588528:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761324.588535:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761324.588575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493934720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761324.588604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761324.588607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761324.588608:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761324.588612:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1329 00000100:00080000:1.0:1789761324.588614:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761324.588616:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761324.588625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761324.588634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493934848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761324.588649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761324.588656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761324.588659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761324.588664:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1329 00000100:00080000:1.0:1789761324.588670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761324.588675:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761324.588683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761324.588699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493934976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761324.588702:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493934720:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761324.588725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761324.588730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761324.588732:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761324.588740:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1329 00000100:00080000:1.0:1789761324.588750:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761324.588757:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761324.588763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761324.588779:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761324.588784:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493935104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761324.588826:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761324/real 1789761324] req@ffff8eb65016c380 x1876699493934720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761324.588841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493934720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761324.588854:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761324.588857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761324.588860:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493934720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761324.588866:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761324.588887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761324.588901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761324/real 1789761324] req@ffff8eb65016d500 x1876699493934848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761324.588914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493934848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761324.588922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761324.588925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761324.588929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934848:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761324.588954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934976:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761324.588991:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761324.589001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761324/real 1789761324] req@ffff8eb65016fb80 x1876699493934976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761324.589009:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493934976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761324.589015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761324.589017:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761324.589020:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493934976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761324.589026:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493935104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761324.589046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761324.589058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761324/real 1789761324] req@ffff8eb65016e300 x1876699493935104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761324.589072:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493935104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761324.589080:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761324.589083:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761324.589085:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493935104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761325.292560:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761326.316582:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761327.340573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761328.364499:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761329.388555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761329.708534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761329.708548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761329.708553:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761329.708568:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1334 00000100:00080000:2.0:1789761329.708584:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761329.708600:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761329.708652:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493935232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761329.708682:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761329.708686:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761329.708689:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761329.708694:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1334 00000100:00080000:2.0:1789761329.708700:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761329.708705:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761329.708715:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493935232:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761329.708718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761329.708738:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493935360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761329.708751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761329.708756:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761329.708759:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761329.708763:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1334 00000100:00080000:2.0:1789761329.708768:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761329.708773:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761329.708780:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761329.708787:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761329.708796:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493935488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761329.708811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761329.708815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761329.708817:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761329.708823:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1334 00000100:00080000:2.0:1789761329.708828:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761329.708831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761329/real 1789761329] req@ffff8eb650f81500 x1876699493935232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761329.708837:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761329.708850:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761329.708859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493935232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761329.708868:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493935616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761329.708872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761329.708877:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761329.708883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493935232:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761329.708914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761329.708951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761329.708962:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761329/real 1789761329] req@ffff8eb650f80380 x1876699493935360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761329.708971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493935360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761329.708977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761329.708979:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761329.708981:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761329.708985:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935488:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761329.708999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761329.709005:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761329/real 1789761329] req@ffff8eb650f80000 x1876699493935488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761329.709013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493935488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761329.709018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761329.709020:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761329.709022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935488:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761329.709025:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493935616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761329.709036:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761329.709042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761329/real 1789761329] req@ffff8eb650f80a80 x1876699493935616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761329.709055:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493935616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761329.709060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761329.709061:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761329.709063:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493935616:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761330.412530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761331.436578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761332.460558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761333.484601:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761334.508538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761334.828558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761334.828569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761334.828573:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761334.828585:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1339 00000100:00080000:1.0:1789761334.828596:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761334.828604:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761334.828650:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493935744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761334.828675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761334.828678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761334.828683:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761334.828687:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1339 00000100:00080000:1.0:1789761334.828690:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761334.828695:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761334.828707:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761334.828723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493935872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761334.828736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761334.828742:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761334.828746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493935744:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761334.828746:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761334.828754:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1339 00000100:00080000:1.0:1789761334.828758:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761334.828762:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761334.828767:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761334.828779:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493936000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761334.828787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761334.828790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761334.828792:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761334.828795:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1339 00000100:00080000:1.0:1789761334.828797:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761334.828800:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761334.828803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761334.828810:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493936128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761334.828837:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761334.828890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761334/real 1789761334] req@ffff8eb65016ce00 x1876699493935744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761334.828926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493935744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761334.828940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761334.828945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761334.828950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493935744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761334.828980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761334.829005:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761334.829018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761334/real 1789761334] req@ffff8eb65016c700 x1876699493935872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761334.829032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493935872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761334.829049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761334.829053:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761334.829057:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493935872:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761334.829064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936000:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761334.829084:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761334.829095:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761334/real 1789761334] req@ffff8eb65016df80 x1876699493936000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761334.829108:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493936000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761334.829118:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761334.829121:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761334.829125:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936000:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761334.829132:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493936128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761334.829150:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761334.829161:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761334/real 1789761334] req@ffff8eb65016dc00 x1876699493936128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761334.829174:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493936128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761334.829183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761334.829186:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761334.829190:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493936128:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761335.532615:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761336.556596:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761337.580564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761338.604543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761339.628567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761339.948479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761339.948507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761339.948511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761339.948523:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1344 00000100:00080000:1.0:1789761339.948535:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761339.948544:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761339.948592:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493936256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761339.948619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761339.948622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761339.948624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761339.948628:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1344 00000100:00080000:1.0:1789761339.948630:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761339.948634:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761339.948644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761339.948657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493936384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761339.948682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493936256:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761339.948685:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761339.948693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761339.948695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761339.948699:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1344 00000100:00080000:1.0:1789761339.948702:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761339.948704:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761339.948708:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761339.948719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493936512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761339.948727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761339.948729:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761339.948731:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761339.948734:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1344 00000100:00080000:1.0:1789761339.948751:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761339.948755:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761339.948760:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761339.948769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493936640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761339.948830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761339.948873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761339.948904:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493936640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761340.652537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761341.676604:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761342.700557:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761343.020702:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761343.020749:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761343.020766:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761343.020810:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761343.020824:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761343.020848:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761343.020873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761339/real 1789761343] req@ffff8eb65016f480 x1876699493936384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.020906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493936384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.020921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.020929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761343.020934:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761343.020942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761339/real 1789761343] req@ffff8eb65016ed80 x1876699493936512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.020956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493936512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.020965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.020969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761343.020973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761343.020980:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761339/real 1789761343] req@ffff8eb65016d180 x1876699493936640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.020993:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493936640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.021000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.021004:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761343.021008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493936640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761343.021015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761339/real 1789761343] req@ffff8eb65016c000 x1876699493936256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.021028:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493936256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.021035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.021040:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761343.021043:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493936256:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789761343.724559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761344.748540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761345.068521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761345.068532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761345.068536:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761345.068547:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1349 00000100:00080000:1.0:1789761345.068559:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761345.068566:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761345.068600:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493936768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761345.068622:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761345.068627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761345.068628:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761345.068633:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1349 00000100:00080000:1.0:1789761345.068637:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761345.068642:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761345.068651:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761345.068653:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493936768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761345.068665:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493936896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761345.068674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761345.068678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761345.068680:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761345.068683:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1349 00000100:00080000:1.0:1789761345.068686:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761345.068689:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761345.068694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761345.068704:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493937024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761345.068710:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761345.068714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761345.068717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761345.068719:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761345.068722:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1349 00000100:00080000:1.0:1789761345.068736:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761345.068740:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761345.068745:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761345.068746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761345/real 1789761345] req@ffff8eb65016f100 x1876699493936768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761345.068755:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493937152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761345.068776:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493936768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761345.068794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761345.068798:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761345.068803:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493936768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761345.068840:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761345.068866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761345.068879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761345/real 1789761345] req@ffff8eb65016ca80 x1876699493936896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761345.068891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493936896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761345.068900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761345.068904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761345.068907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493936896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761345.068913:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937024:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761345.068934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761345.068946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761345/real 1789761345] req@ffff8eb65016c000 x1876699493937024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761345.068957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493937024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761345.068964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761345.068966:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761345.068970:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937024:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761345.068976:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493937152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761345.068995:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761345.069006:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761345/real 1789761345] req@ffff8eb65016d180 x1876699493937152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761345.069018:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493937152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761345.069026:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761345.069029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761345.069032:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493937152:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761345.772564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761346.796561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761347.820584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761348.844519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761349.868503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761350.188489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761350.188497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761350.188500:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761350.188509:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1354 00000100:00080000:2.0:1789761350.188520:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761350.188528:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761350.188565:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493937280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761350.188593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761350.188597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761350.188599:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761350.188604:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1354 00000100:00080000:2.0:1789761350.188609:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761350.188614:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761350.188623:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493937280:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761350.188624:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761350.188642:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493937408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761350.188654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761350.188657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761350.188659:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761350.188661:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1354 00000100:00080000:2.0:1789761350.188664:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761350.188668:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761350.188672:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761350.188680:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761350.188683:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493937536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761350.188692:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761350.188695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761350.188697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761350.188699:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1354 00000100:00080000:2.0:1789761350.188702:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761350.188704:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761350.188708:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761350.188716:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493937664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761350.188723:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761350/real 1789761350] req@ffff8eb650f83800 x1876699493937280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.188746:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493937280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.188758:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.188761:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761350.188764:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493937280:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761350.188795:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761350.188815:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761350.188826:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761350/real 1789761350] req@ffff8eb650f81180 x1876699493937408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.188835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493937408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.188841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.188844:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761350.188846:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761350.188850:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937536:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761350.188862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761350.188868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761350/real 1789761350] req@ffff8eb650f81f80 x1876699493937536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.188876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493937536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.188881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.188882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761350.188884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761350.188888:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493937664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761350.188898:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761350.188904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761350/real 1789761350] req@ffff8eb650f83b80 x1876699493937664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761350.188911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493937664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761350.188917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761350.188919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761350.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493937664:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761350.892524:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761351.916550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761352.940544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761353.964513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761354.988633:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761355.308470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761355.308478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761355.308482:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761355.308489:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1359 00000100:00080000:2.0:1789761355.308504:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761355.308510:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761355.308543:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493937792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761355.308572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761355.308576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761355.308578:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761355.308583:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1359 00000100:00080000:2.0:1789761355.308587:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761355.308592:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761355.308601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761355.308608:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493937792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761355.308615:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493937920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761355.308625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761355.308628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761355.308630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761355.308632:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1359 00000100:00080000:2.0:1789761355.308635:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761355.308638:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761355.308642:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761355.308657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493938048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761355.308665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761355.308669:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761355.308671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761355.308674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761355.308680:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1359 00000100:00080000:2.0:1789761355.308685:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761355.308689:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761355.308695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761355.308709:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493938176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761355.308713:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761355/real 1789761355] req@ffff8eb650f83480 x1876699493937792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761355.308757:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493937792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761355.308770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761355.308775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761355.308779:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493937792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761355.308804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761355.308824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761355.308833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761355/real 1789761355] req@ffff8eb650f80e00 x1876699493937920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761355.308842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493937920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761355.308848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761355.308850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761355.308853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493937920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761355.308857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938048:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761355.308869:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761355.308875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761355/real 1789761355] req@ffff8eb650f80700 x1876699493938048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761355.308883:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493938048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761355.308888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761355.308890:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761355.308892:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938048:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761355.308895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493938176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761355.308905:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761355.308911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761355/real 1789761355] req@ffff8eb650f82a00 x1876699493938176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761355.308918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493938176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761355.308923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761355.308925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761355.308927:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493938176:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761356.012534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761357.036542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761358.060540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761359.084590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761360.108583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761360.428567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761360.428577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761360.428581:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761360.428593:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1364 00000100:00080000:2.0:1789761360.428607:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761360.428617:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761360.428672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493938304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761360.428695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761360.428715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761360.428717:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761360.428722:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1364 00000100:00080000:2.0:1789761360.428726:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761360.428730:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761360.428740:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761360.428746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493938304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761360.428757:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493938432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761360.428769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761360.428774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761360.428775:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761360.428779:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1364 00000100:00080000:2.0:1789761360.428782:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761360.428785:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761360.428789:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761360.428797:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493938560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761360.428806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761360.428810:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761360.428811:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761360.428817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761360.428818:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1364 00000100:00080000:2.0:1789761360.428824:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761360.428828:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761360.428834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761360.428847:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493938688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761360.428862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761360/real 1789761360] req@ffff8eb650f81c00 x1876699493938304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761360.428880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493938304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761360.428888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761360.428892:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761360.428895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493938304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761360.428914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761360.428929:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761360.428937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761360/real 1789761360] req@ffff8eb650f82680 x1876699493938432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761360.428945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493938432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761360.428950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761360.428952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761360.428954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761360.428958:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938560:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761360.428969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761360.428975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761360/real 1789761360] req@ffff8eb650f82d80 x1876699493938560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761360.428983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493938560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761360.428988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761360.428990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761360.428992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938560:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761360.428996:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493938688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761360.429006:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761360.429012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761360/real 1789761360] req@ffff8eb650f81880 x1876699493938688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761360.429019:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493938688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761360.429023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761360.429025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761360.429027:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493938688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761361.132521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761362.156502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761363.180513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761364.204545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761365.228578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761365.548506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761365.548517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761365.548521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761365.548532:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1370 00000100:00080000:1.0:1789761365.548544:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761365.548552:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761365.548602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493938816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761365.548643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761365.548647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761365.548648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761365.548655:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1370 00000100:00080000:1.0:1789761365.548658:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761365.548675:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761365.548686:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761365.548698:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493938816:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761365.548701:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493938944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761365.548712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761365.548715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761365.548716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761365.548719:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1370 00000100:00080000:1.0:1789761365.548722:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761365.548725:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761365.548729:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761365.548740:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493939072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761365.548747:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761365.548750:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761365.548752:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761365.548754:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1370 00000100:00080000:1.0:1789761365.548758:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761365.548761:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761365.548765:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761365.548774:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493939200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761365.548778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761365.548824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761365/real 1789761365] req@ffff8eb65016ed80 x1876699493938816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761365.548839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493938816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761365.548849:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761365.548852:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761365.548856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493938816:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761365.548878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761365.548902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761365.548912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761365/real 1789761365] req@ffff8eb65016f480 x1876699493938944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761365.548921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493938944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761365.548927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761365.548929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761365.548931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493938944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761365.548936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939072:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761365.548948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761365.548954:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761365/real 1789761365] req@ffff8eb65016dc00 x1876699493939072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761365.548962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493939072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761365.548967:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761365.548969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761365.548971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761365.548975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493939200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761365.548986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761365.548992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761365/real 1789761365] req@ffff8eb65016df80 x1876699493939200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761365.548999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493939200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761365.549005:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761365.549007:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761365.549010:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493939200:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761366.252552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761367.276546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761368.300515:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761369.324561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761370.348520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761370.668474:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761370.668484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761370.668489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761370.668500:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1375 00000100:00080000:1.0:1789761370.668511:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761370.668520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761370.668555:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493939328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761370.668582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761370.668585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761370.668587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761370.668591:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1375 00000100:00080000:1.0:1789761370.668594:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761370.668598:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761370.668608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761370.668622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493939456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761370.668637:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493939328:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761370.668657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761370.668661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761370.668664:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761370.668668:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1375 00000100:00080000:1.0:1789761370.668673:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761370.668685:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761370.668691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761370.668705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493939584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761370.668708:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761370.668721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761370.668725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761370.668727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761370.668732:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1375 00000100:00080000:1.0:1789761370.668736:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761370.668740:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761370.668745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761370.668748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761370/real 1789761370] req@ffff8eb65016c700 x1876699493939328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761370.668760:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493939712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761370.668772:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493939328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761370.668791:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761370.668797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761370.668802:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493939328:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761370.668823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761370.668844:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761370.668853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761370/real 1789761370] req@ffff8eb65016ce00 x1876699493939456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761370.668862:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493939456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761370.668868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761370.668870:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761370.668872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939456:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761370.668876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939584:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761370.668888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761370.668894:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761370/real 1789761370] req@ffff8eb65016e300 x1876699493939584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761370.668902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493939584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761370.668907:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761370.668909:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761370.668911:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939584:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761370.668915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493939712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761370.668938:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761370.668949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761370/real 1789761370] req@ffff8eb65016fb80 x1876699493939712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761370.668970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493939712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761370.668979:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761370.668982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761370.668985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493939712:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761371.372510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761372.396526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761373.420524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761374.444553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761375.468534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761375.788461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761375.788469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761375.788472:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761375.788481:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1380 00000100:00080000:1.0:1789761375.788490:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761375.788495:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761375.788525:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493939840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761375.788547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761375.788549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761375.788551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761375.788554:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1380 00000100:00080000:1.0:1789761375.788556:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761375.788558:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761375.788565:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761375.788574:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493939968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761375.788581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761375.788584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761375.788585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761375.788602:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1380 00000100:00080000:1.0:1789761375.788606:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761375.788609:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493939840:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761375.788612:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761375.788622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761375.788631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493940096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761375.788639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761375.788642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761375.788643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761375.788645:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1380 00000100:00080000:1.0:1789761375.788649:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761375.788652:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761375.788655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761375.788663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493940224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761375.788686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761375.788724:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761375/real 1789761375] req@ffff8eb65016d500 x1876699493939840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761375.788742:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493939840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761375.788752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761375.788756:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761375.788760:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493939840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761375.788772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761375.788795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761375.788806:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761375/real 1789761375] req@ffff8eb65016c380 x1876699493939968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761375.788826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493939968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761375.788835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761375.788839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761375.788842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493939968:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761375.788871:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940096:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761375.788892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761375.788902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761375/real 1789761375] req@ffff8eb65016f800 x1876699493940096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761375.788913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493940096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761375.788922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761375.788925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761375.788929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761375.788935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493940224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761375.788951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761375.788960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761375/real 1789761375] req@ffff8eb65016e680 x1876699493940224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761375.788971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493940224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761375.788979:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761375.788982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761375.788986:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493940224:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761376.492601:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761377.516561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761378.540556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761379.564590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761380.588596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761380.908476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761380.908484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761380.908487:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761380.908494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1385 00000100:00080000:1.0:1789761380.908503:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761380.908508:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761380.908542:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493940352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761380.908581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761380.908585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761380.908587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761380.908592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1385 00000100:00080000:1.0:1789761380.908597:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761380.908601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761380.908610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761380.908625:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493940352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761380.908625:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493940480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761380.908642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761380.908645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761380.908646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761380.908649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1385 00000100:00080000:1.0:1789761380.908652:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761380.908655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761380.908659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761380.908668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493940608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761380.908675:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761380.908678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761380.908679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761380.908682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1385 00000100:00080000:1.0:1789761380.908685:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761380.908686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761380.908694:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761380.908701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761380.908716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493940736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761380.908727:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761380/real 1789761380] req@ffff8eb65016ea00 x1876699493940352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761380.908750:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493940352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761380.908762:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761380.908767:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761380.908772:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493940352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761380.908797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761380.908819:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761380.908829:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761380/real 1789761380] req@ffff8eb65016d880 x1876699493940480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761380.908838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493940480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761380.908844:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761380.908846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761380.908848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761380.908852:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940608:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761380.908865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761380.908871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761380/real 1789761380] req@ffff8eb65016e680 x1876699493940608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761380.908878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493940608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761380.908883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761380.908885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761380.908887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940608:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761380.908890:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493940736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761380.908900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761380.908906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761380/real 1789761380] req@ffff8eb65016f800 x1876699493940736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761380.908913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493940736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761380.908918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761380.908920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761380.908921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493940736:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761381.612515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761382.636570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761383.660516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761384.684568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761385.708541:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761386.028464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761386.028473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761386.028476:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761386.028484:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1390 00000100:00080000:1.0:1789761386.028493:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761386.028499:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761386.028543:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493940864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761386.028572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761386.028575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761386.028576:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761386.028579:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1390 00000100:00080000:1.0:1789761386.028582:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761386.028584:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761386.028592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761386.028601:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493940992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761386.028608:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493940864:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761386.028613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761386.028616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761386.028618:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761386.028621:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1390 00000100:00080000:1.0:1789761386.028623:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761386.028626:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761386.028629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761386.028638:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493941120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761386.028645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761386.028648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761386.028649:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761386.028651:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1390 00000100:00080000:1.0:1789761386.028655:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761386.028658:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761386.028661:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761386.028667:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761386.028671:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493941248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761386.028705:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761386/real 1789761386] req@ffff8eb65016c380 x1876699493940864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761386.028719:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493940864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761386.028727:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761386.028730:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761386.028733:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493940864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761386.028750:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761386.028764:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761386.028772:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761386/real 1789761386] req@ffff8eb65016d500 x1876699493940992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761386.028779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493940992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761386.028786:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761386.028788:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761386.028790:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493940992:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761386.028793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941120:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761386.028803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761386.028809:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761386/real 1789761386] req@ffff8eb65016fb80 x1876699493941120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761386.028816:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493941120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761386.028828:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761386.028831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761386.028835:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941120:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761386.028841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493941248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761386.028859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761386.028882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761386/real 1789761386] req@ffff8eb65016e300 x1876699493941248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761386.028901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493941248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761386.028910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761386.028913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761386.028917:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493941248:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761386.732534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761387.756527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761388.780557:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761389.804538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761390.828567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761391.148503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761391.148512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761391.148516:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761391.148525:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1395 00000100:00080000:1.0:1789761391.148534:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761391.148540:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761391.148578:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493941376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761391.148603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761391.148606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761391.148608:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761391.148612:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1395 00000100:00080000:1.0:1789761391.148615:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761391.148618:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761391.148625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761391.148635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493941504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761391.148643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761391.148646:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493941376:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761391.148647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761391.148651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761391.148657:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1395 00000100:00080000:1.0:1789761391.148659:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761391.148661:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761391.148664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761391.148673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493941632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761391.148680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761391.148683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761391.148685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761391.148688:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1395 00000100:00080000:1.0:1789761391.148690:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761391.148692:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761391.148696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761391.148703:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493941760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761391.148709:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761391.148752:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761391/real 1789761391] req@ffff8eb65016ce00 x1876699493941376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761391.148767:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493941376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761391.148775:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761391.148777:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761391.148780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493941376:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761391.148799:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761391.148816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761391.148824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761391/real 1789761391] req@ffff8eb65016c700 x1876699493941504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761391.148833:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493941504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761391.148838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761391.148841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761391.148843:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761391.148848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941632:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761391.148859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761391.148865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761391/real 1789761391] req@ffff8eb65016df80 x1876699493941632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761391.148875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493941632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761391.148884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761391.148888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761391.148891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493941632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761391.148897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493941760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761391.148917:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761391.148930:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761391/real 1789761391] req@ffff8eb65016dc00 x1876699493941760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761391.148943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493941760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761391.148948:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761391.148951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761391.148953:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493941760:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761391.852539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761392.876552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761393.900557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761394.924544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761395.948539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761396.268505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761396.268515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761396.268519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761396.268528:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1400 00000100:00080000:3.0:1789761396.268538:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761396.268545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761396.268581:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493941888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761396.268604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761396.268608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761396.268609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761396.268612:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1400 00000100:00080000:3.0:1789761396.268615:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761396.268618:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761396.268626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761396.268636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493942016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761396.268643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761396.268646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761396.268648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761396.268650:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1400 00000100:00080000:3.0:1789761396.268653:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761396.268655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761396.268658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761396.268665:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493941888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761396.268666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493942144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761396.268680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761396.268683:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761396.268684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761396.268686:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1400 00000100:00080000:3.0:1789761396.268688:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761396.268690:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761396.268693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761396.268700:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493942272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761396.268747:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761396.268789:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761396/real 1789761396] req@ffff8eb64514ad80 x1876699493941888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761396.268809:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493941888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761396.268821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761396.268826:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761396.268830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493941888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761396.268838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761396.268864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761396.268879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761396/real 1789761396] req@ffff8eb64514b480 x1876699493942016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761396.268892:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493942016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761396.268901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761396.268905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761396.268909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761396.268931:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942144:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761396.268948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761396.268957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761396/real 1789761396] req@ffff8eb645148700 x1876699493942144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761396.268965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493942144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761396.268975:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761396.268977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761396.268980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942144:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761396.268983:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493942272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761396.268994:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761396.269000:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761396/real 1789761396] req@ffff8eb645149180 x1876699493942272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761396.269007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493942272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761396.269012:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761396.269014:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761396.269016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493942272:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761396.972533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761397.996585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761399.020555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761400.044560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761401.068521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761401.388499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761401.388509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761401.388519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761401.388531:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1405 00000100:00080000:3.0:1789761401.388543:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761401.388551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761401.388594:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493942400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761401.388619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761401.388621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761401.388623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761401.388626:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1405 00000100:00080000:3.0:1789761401.388629:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761401.388631:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761401.388640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761401.388650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493942528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761401.388658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761401.388661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761401.388663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761401.388666:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1405 00000100:00080000:3.0:1789761401.388668:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761401.388671:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761401.388674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761401.388682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493942656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761401.388691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761401.388695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761401.388700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761401.388702:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493942400:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761401.388706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1405 00000100:00080000:3.0:1789761401.388717:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761401.388722:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761401.388726:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761401.388735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493942784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761401.388778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761401.388815:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761401/real 1789761401] req@ffff8eb645149180 x1876699493942400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761401.388829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493942400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761401.388838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761401.388841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761401.388844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493942400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761401.388851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761401.388865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761401.388873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761401/real 1789761401] req@ffff8eb645148700 x1876699493942528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761401.388883:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493942528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761401.388891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761401.388895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761401.388898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942528:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761401.388924:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942656:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761401.388946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761401.388963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761401/real 1789761401] req@ffff8eb64514b480 x1876699493942656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761401.388971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493942656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761401.388977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761401.388979:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761401.388981:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493942656:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761401.388985:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493942784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761401.388999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761401.389005:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761401/real 1789761401] req@ffff8eb64514ad80 x1876699493942784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761401.389013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493942784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761401.389018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761401.389020:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761401.389022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493942784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761402.092542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761403.116499:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761404.140550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761405.164628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761406.188570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761406.508461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761406.508469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761406.508473:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761406.508481:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1411 00000100:00080000:3.0:1789761406.508490:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761406.508496:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761406.508526:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493942912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761406.508548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761406.508551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761406.508553:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761406.508555:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1411 00000100:00080000:3.0:1789761406.508557:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761406.508560:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761406.508567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761406.508575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493943040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761406.508583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761406.508586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761406.508587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761406.508590:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1411 00000100:00080000:3.0:1789761406.508591:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761406.508593:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761406.508597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761406.508607:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493943168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761406.508609:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493942912:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761406.508628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761406.508631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761406.508632:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761406.508635:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1411 00000100:00080000:3.0:1789761406.508637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761406.508640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761406.508644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761406.508651:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493943296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761406.508711:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761406.508762:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761406.508787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493943296:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761407.212523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761408.236519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761409.260546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761409.580598:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789761409.580633:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761409.580647:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761409.580685:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761409.580697:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761409.580713:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789761409.580724:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761406/real 1789761409] req@ffff8eb645149880 x1876699493943168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761409.580757:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493943168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761409.580770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761409.580776:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761409.580780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761409.580789:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761406/real 1789761409] req@ffff8eb64514b100 x1876699493943296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761409.580804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493943296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761409.580814:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761409.580817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761409.580821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493943296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761409.580828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761406/real 1789761409] req@ffff8eb64514bb80 x1876699493942912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761409.580841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493942912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761409.580850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761409.580853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761409.580857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493942912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761409.580864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761406/real 1789761409] req@ffff8eb645148000 x1876699493943040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761409.580877:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493943040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761409.580884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761409.580888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761409.580891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761410.284548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761411.308524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761411.628468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761411.628476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761411.628479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761411.628487:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1416 00000100:00080000:3.0:1789761411.628496:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761411.628503:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761411.628544:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493943424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761411.628574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761411.628579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761411.628581:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761411.628586:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1416 00000100:00080000:3.0:1789761411.628589:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761411.628595:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761411.628604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761411.628616:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493943424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761411.628623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493943552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761411.628637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761411.628642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761411.628644:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761411.628647:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1416 00000100:00080000:3.0:1789761411.628651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761411.628654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761411.628660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761411.628675:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493943680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761411.628685:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761411.628689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761411.628696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761411.628699:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761411.628704:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1416 00000100:00080000:3.0:1789761411.628709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761411.628714:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761411.628720:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761411.628735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493943808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761411.628738:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761411/real 1789761411] req@ffff8eb64514b800 x1876699493943424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761411.628775:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493943424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761411.628788:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761411.628793:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761411.628798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493943424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761411.628823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761411.628850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761411.628865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761411/real 1789761411] req@ffff8eb64514a300 x1876699493943552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761411.628880:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493943552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761411.628889:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761411.628893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761411.628896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943552:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761411.628903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943680:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761411.628925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761411.628938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761411/real 1789761411] req@ffff8eb645148e00 x1876699493943680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761411.628949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493943680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761411.628964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761411.628968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761411.628972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493943680:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761411.628978:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493943808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761411.629000:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761411.629012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761411/real 1789761411] req@ffff8eb645148380 x1876699493943808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761411.629025:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493943808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761411.629034:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761411.629038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761411.629041:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493943808:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761412.332518:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761413.356518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761414.380523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761415.404598:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761416.428554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761416.748469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761416.748477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761416.748480:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761416.748488:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1421 00000100:00080000:3.0:1789761416.748497:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761416.748504:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761416.748538:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493943936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761416.748560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761416.748562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761416.748564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761416.748567:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1421 00000100:00080000:3.0:1789761416.748569:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761416.748572:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761416.748580:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761416.748592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493944064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761416.748600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761416.748602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761416.748603:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761416.748606:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1421 00000100:00080000:3.0:1789761416.748607:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761416.748610:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761416.748613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761416.748623:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493943936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761416.748627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493944192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761416.748644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761416.748646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761416.748647:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761416.748652:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1421 00000100:00080000:3.0:1789761416.748654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761416.748657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761416.748660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761416.748672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493944320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761416.748721:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761416.748766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761416/real 1789761416] req@ffff8eb645149c00 x1876699493943936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761416.748782:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493943936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761416.748791:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761416.748794:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761416.748798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493943936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761416.748805:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761416.748822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761416.748830:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761416/real 1789761416] req@ffff8eb645149500 x1876699493944064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761416.748838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493944064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761416.748844:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761416.748847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761416.748849:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944064:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761416.748874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944192:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761416.748886:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761416.748892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761416/real 1789761416] req@ffff8eb645148a80 x1876699493944192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761416.748900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493944192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761416.748906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761416.748908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761416.748910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761416.748914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493944320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761416.748925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761416.748931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761416/real 1789761416] req@ffff8eb64514a680 x1876699493944320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761416.748939:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493944320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761416.748945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761416.748946:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761416.748948:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493944320:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761417.452590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761418.476571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761419.500628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761420.524547:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761421.548630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761421.868472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761421.868480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761421.868484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761421.868492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1426 00000100:00080000:3.0:1789761421.868506:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761421.868512:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761421.868543:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493944448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761421.868569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761421.868573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761421.868575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761421.868580:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1426 00000100:00080000:3.0:1789761421.868584:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761421.868589:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761421.868599:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761421.868615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493944576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761421.868629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761421.868631:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493944448:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761421.868635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761421.868643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761421.868648:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1426 00000100:00080000:3.0:1789761421.868651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761421.868655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761421.868660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761421.868672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493944704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761421.868682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761421.868686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761421.868688:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761421.868691:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1426 00000100:00080000:3.0:1789761421.868717:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761421.868720:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761421.868723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761421.868730:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761421.868734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493944832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761421.868781:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761421/real 1789761421] req@ffff8eb64514a680 x1876699493944448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761421.868803:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493944448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761421.868813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761421.868816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761421.868820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493944448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761421.868844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761421.868862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761421.868870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761421/real 1789761421] req@ffff8eb645148a80 x1876699493944576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761421.868879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493944576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761421.868891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761421.868893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761421.868896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761421.868900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944704:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761421.868912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761421.868919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761421/real 1789761421] req@ffff8eb645149500 x1876699493944704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761421.868926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493944704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761421.868932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761421.868934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761421.868936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493944704:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761421.868940:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493944832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761421.868951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761421.868957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761421/real 1789761421] req@ffff8eb645149c00 x1876699493944832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761421.868965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493944832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761421.868969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761421.868971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761421.868974:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493944832:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761422.572602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761423.596611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761424.620608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761425.644579:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761426.668531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761426.988467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761426.988474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761426.988477:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761426.988485:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1431 00000100:00080000:3.0:1789761426.988494:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761426.988500:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761426.988531:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493944960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761426.988552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761426.988556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761426.988565:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761426.988572:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1431 00000100:00080000:3.0:1789761426.988576:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761426.988580:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761426.988589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761426.988596:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493944960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761426.988601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493945088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761426.988612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761426.988615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761426.988617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761426.988620:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1431 00000100:00080000:3.0:1789761426.988623:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761426.988626:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761426.988630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761426.988639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493945216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761426.988648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761426.988650:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761426.988652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761426.988654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761426.988657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1431 00000100:00080000:3.0:1789761426.988660:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761426.988663:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761426.988667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761426.988676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493945344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761426.988698:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761426/real 1789761426] req@ffff8eb645148380 x1876699493944960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761426.988716:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493944960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761426.988727:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761426.988730:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761426.988739:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493944960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761426.988759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761426.988777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761426.988786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761426/real 1789761426] req@ffff8eb645148e00 x1876699493945088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761426.988794:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493945088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761426.988800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761426.988803:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761426.988807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945088:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761426.988813:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945216:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761426.988833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761426.988846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761426/real 1789761426] req@ffff8eb64514a300 x1876699493945216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761426.988860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493945216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761426.988867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761426.988869:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761426.988871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945216:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761426.988875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493945344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761426.988890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761426.988897:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761426/real 1789761426] req@ffff8eb64514b800 x1876699493945344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761426.988905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493945344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761426.988910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761426.988912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761426.988914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493945344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761427.692556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761428.716502:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761429.740521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761430.764512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761431.788557:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761432.108489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761432.108497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761432.108500:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761432.108509:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1436 00000100:00080000:3.0:1789761432.108518:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761432.108524:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761432.108555:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493945472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761432.108576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761432.108579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761432.108580:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761432.108583:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1436 00000100:00080000:3.0:1789761432.108585:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761432.108588:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761432.108595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761432.108604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493945600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761432.108612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761432.108614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761432.108616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761432.108618:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1436 00000100:00080000:3.0:1789761432.108620:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761432.108622:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761432.108626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761432.108629:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493945472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761432.108637:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493945728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761432.108647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761432.108651:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761432.108653:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761432.108657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1436 00000100:00080000:3.0:1789761432.108661:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761432.108666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761432.108671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761432.108685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493945856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761432.108695:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761432.108745:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761432/real 1789761432] req@ffff8eb645148000 x1876699493945472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761432.108764:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493945472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761432.108777:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761432.108781:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761432.108785:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493945472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761432.108793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761432.108818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761432.108832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761432/real 1789761432] req@ffff8eb64514bb80 x1876699493945600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761432.108842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493945600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761432.108848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761432.108850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761432.108853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761432.108871:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945728:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761432.108885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761432.108893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761432/real 1789761432] req@ffff8eb64514b100 x1876699493945728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761432.108906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493945728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761432.108914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761432.108917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761432.108920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493945728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761432.108927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493945856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761432.108948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761432.108961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761432/real 1789761432] req@ffff8eb645149880 x1876699493945856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761432.108974:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493945856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761432.108983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761432.108986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761432.108989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493945856:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761432.812541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761433.836512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761434.860544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761435.884519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761436.908558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761437.228465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761437.228473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761437.228477:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761437.228485:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1441 00000100:00080000:3.0:1789761437.228495:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761437.228500:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761437.228531:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493945984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761437.228552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761437.228554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761437.228555:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761437.228558:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1441 00000100:00080000:3.0:1789761437.228560:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761437.228563:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761437.228571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761437.228582:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493946112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761437.228594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761437.228597:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493945984:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761437.228599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761437.228603:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761437.228610:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1441 00000100:00080000:3.0:1789761437.228614:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761437.228619:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761437.228625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761437.228638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493946240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761437.228650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761437.228653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761437.228655:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761437.228660:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1441 00000100:00080000:3.0:1789761437.228664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761437.228669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:2.0:1789761437.228674:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789761437.228678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761437.228694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493946368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761437.228724:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761437/real 1789761437] req@ffff8eb64514ad80 x1876699493945984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761437.228745:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493945984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761437.228757:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761437.228761:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761437.228766:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493945984:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761437.228788:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761437.228811:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761437.228824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761437/real 1789761437] req@ffff8eb64514b480 x1876699493946112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761437.228838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493946112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761437.228847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761437.228851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761437.228855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946112:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761437.228862:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946240:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761437.228880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761437.228891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761437/real 1789761437] req@ffff8eb645148700 x1876699493946240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761437.228904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493946240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761437.228913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761437.228917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761437.228920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946240:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761437.228937:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493946368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761437.228957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761437.228968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761437/real 1789761437] req@ffff8eb645149180 x1876699493946368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761437.228982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493946368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761437.228998:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761437.229002:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761437.229006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493946368:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761437.932536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761438.956505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761439.980549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761441.004559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761442.028522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761442.348525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761442.348538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761442.348544:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761442.348557:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1446 00000100:00080000:1.0:1789761442.348571:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761442.348580:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761442.348623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493946496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761442.348654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761442.348659:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761442.348661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761442.348667:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1446 00000100:00080000:1.0:1789761442.348680:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761442.348686:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761442.348700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761442.348704:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493946496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761442.348716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493946624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761442.348728:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761442.348732:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761442.348733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761442.348737:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1446 00000100:00080000:1.0:1789761442.348743:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761442.348747:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761442.348752:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761442.348760:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493946752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761442.348770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789761442.348777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761442.348779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761442.348782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761442.348788:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1446 00000100:00080000:1.0:1789761442.348794:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761442.348798:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761442.348802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761442.348814:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493946880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761442.348823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761442/real 1789761442] req@ffff8eb65016f480 x1876699493946496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761442.348842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493946496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761442.348851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761442.348855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761442.348858:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493946496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761442.348891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761442.348909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761442.348917:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761442/real 1789761442] req@ffff8eb65016ed80 x1876699493946624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761442.348926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493946624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761442.348932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761442.348934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761442.348936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946624:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761442.348942:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946752:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761442.348953:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761442.348960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761442/real 1789761442] req@ffff8eb65016d180 x1876699493946752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761442.348967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493946752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761442.348972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761442.348974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761442.348976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493946752:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761442.348981:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493946880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761442.348992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761442.348998:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761442/real 1789761442] req@ffff8eb65016c000 x1876699493946880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761442.349005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493946880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761442.349010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761442.349012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761442.349014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493946880:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761443.052556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761444.076567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761445.100576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761446.124539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761447.148537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761447.468552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761447.468568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761447.468573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761447.468588:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1451 00000100:00080000:1.0:1789761447.468604:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761447.468614:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761447.468662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493947008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761447.468692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761447.468696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761447.468699:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761447.468703:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1451 00000100:00080000:1.0:1789761447.468707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761447.468712:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761447.468723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761447.468736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493947136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761447.468745:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493947008:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761447.468749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761447.468759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761447.468764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761447.468768:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1451 00000100:00080000:1.0:1789761447.468770:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761447.468773:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761447.468776:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761447.468788:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493947264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761447.468796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761447.468799:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761447.468801:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761447.468804:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1451 00000100:00080000:1.0:1789761447.468806:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761447.468810:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761447.468814:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761447.468822:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493947392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761447.468834:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761447.468871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761447/real 1789761447] req@ffff8eb65016ca80 x1876699493947008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761447.468893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493947008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761447.468905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761447.468910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761447.468914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493947008:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761447.468941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761447.468968:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761447.468983:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761447/real 1789761447] req@ffff8eb65016f100 x1876699493947136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761447.468998:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493947136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761447.469007:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761447.469011:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761447.469015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947136:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761447.469022:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947264:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761447.469044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761447.469069:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761447/real 1789761447] req@ffff8eb65016c000 x1876699493947264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761447.469082:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493947264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761447.469097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761447.469101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761447.469104:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947264:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761447.469111:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493947392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761447.469135:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761447.469148:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761447/real 1789761447] req@ffff8eb65016d180 x1876699493947392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761447.469161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493947392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761447.469169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761447.469173:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761447.469176:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493947392:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761448.172528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761449.196555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761450.220521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761451.244574:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761452.268555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761452.588492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761452.588500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761452.588503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761452.588512:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1457 00000100:00080000:3.0:1789761452.588520:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761452.588526:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761452.588557:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493947520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761452.588580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761452.588583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761452.588584:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761452.588587:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1457 00000100:00080000:3.0:1789761452.588589:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761452.588592:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761452.588598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761452.588607:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493947648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761452.588614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761452.588617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761452.588618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761452.588620:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1457 00000100:00080000:3.0:1789761452.588622:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761452.588627:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761452.588629:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493947520:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789761452.588631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761452.588641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493947776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761452.588648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761452.588650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761452.588651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761452.588654:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1457 00000100:00080000:3.0:1789761452.588657:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761452.588660:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761452.588663:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761452.588670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493947904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761452.588686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761452.588718:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761452/real 1789761452] req@ffff8eb645149180 x1876699493947520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761452.588731:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493947520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761452.588739:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761452.588741:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761452.588745:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493947520:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761452.588750:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761452.588764:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761452.588771:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761452/real 1789761452] req@ffff8eb645148700 x1876699493947648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761452.588779:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493947648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761452.588785:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761452.588787:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761452.588789:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947648:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761452.588806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947776:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761452.588817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761452.588822:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761452/real 1789761452] req@ffff8eb64514b480 x1876699493947776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761452.588830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493947776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761452.588835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761452.588837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761452.588839:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493947776:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761452.588843:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493947904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761452.588852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761452.588858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761452/real 1789761452] req@ffff8eb64514ad80 x1876699493947904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761452.588866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493947904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761452.588879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761452.588883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761452.588885:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493947904:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761453.292507:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761454.316572:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761455.340531:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761456.364543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761457.388524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761457.708516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761457.708524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761457.708528:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761457.708536:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1462 00000100:00080000:2.0:1789761457.708546:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761457.708551:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761457.708583:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493948032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761457.708606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761457.708609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761457.708610:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761457.708613:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1462 00000100:00080000:2.0:1789761457.708615:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761457.708618:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761457.708627:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761457.708639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493948160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761457.708653:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493948032:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761457.708666:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761457.708670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761457.708672:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761457.708676:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1462 00000100:00080000:2.0:1789761457.708679:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761457.708683:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761457.708687:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761457.708698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493948288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761457.708705:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761457.708708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761457.708710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761457.708714:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761457.708716:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1462 00000100:00080000:2.0:1789761457.708721:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761457.708723:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761457.708727:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761457.708735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493948416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761457.708763:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761457/real 1789761457] req@ffff8eb650f81880 x1876699493948032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761457.708783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493948032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761457.708795:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761457.708799:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761457.708802:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493948032:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761457.708821:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761457.708840:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761457.708849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761457/real 1789761457] req@ffff8eb650f82d80 x1876699493948160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761457.708857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493948160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761457.708862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761457.708864:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761457.708867:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948160:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761457.708870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948288:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761457.708882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761457.708888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761457/real 1789761457] req@ffff8eb650f82680 x1876699493948288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761457.708895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493948288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761457.708901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761457.708902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761457.708904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948288:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761457.708908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493948416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761457.708921:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761457.708931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761457/real 1789761457] req@ffff8eb650f81c00 x1876699493948416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761457.708950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493948416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761457.708959:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761457.708962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761457.708965:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493948416:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761458.412506:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761459.436510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761460.460594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761461.484567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761462.508554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761462.828485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761462.828496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761462.828502:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761462.828512:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1467 00000100:00080000:2.0:1789761462.828524:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761462.828532:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761462.828570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493948544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761462.828599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761462.828603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761462.828605:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761462.828610:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1467 00000100:00080000:2.0:1789761462.828615:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761462.828620:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761462.828629:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493948544:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761462.828630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761462.828649:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493948672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761462.828661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761462.828665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761462.828667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761462.828670:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1467 00000100:00080000:2.0:1789761462.828677:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761462.828682:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761462.828686:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789761462.828688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761462.828705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493948800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761462.828717:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761462.828721:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761462/real 1789761462] req@ffff8eb650f82a00 x1876699493948544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761462.828722:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761462.828726:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761462.828734:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1467 00000100:00080000:2.0:1789761462.828739:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761462.828744:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493948544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761462.828744:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761462.828752:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761462.828761:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761462.828768:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761462.828769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493948928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761462.828774:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493948544:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761462.828800:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761462.828817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761462.828825:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761462/real 1789761462] req@ffff8eb650f80700 x1876699493948672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761462.828834:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493948672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761462.828840:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761462.828842:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761462.828844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948672:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761462.828848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948800:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761462.828859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761462.828865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761462/real 1789761462] req@ffff8eb650f80e00 x1876699493948800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761462.828873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493948800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761462.828881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761462.828884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761462.828888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493948800:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761462.828894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493948928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761462.828920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761462.828933:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761462/real 1789761462] req@ffff8eb650f83480 x1876699493948928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761462.828959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493948928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761462.828968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761462.828971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761462.828976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493948928:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761463.532540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761464.556546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761465.580525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761466.604554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761467.628519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761467.948533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761467.948543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761467.948548:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761467.948558:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1472 00000100:00080000:3.0:1789761467.948569:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761467.948576:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761467.948616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493949056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761467.948646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761467.948650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761467.948653:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761467.948658:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1472 00000100:00080000:3.0:1789761467.948664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761467.948669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761467.948680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761467.948682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493949056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761467.948698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493949184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761467.948711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761467.948715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761467.948717:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761467.948721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1472 00000100:00080000:3.0:1789761467.948725:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761467.948730:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761467.948735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761467.948750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493949312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761467.948755:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761467.948768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761467.948772:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761467.948774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761467.948780:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1472 00000100:00080000:3.0:1789761467.948784:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761467.948789:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761467.948795:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761467.948801:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761467/real 1789761467] req@ffff8eb645149880 x1876699493949056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761467.948812:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493949440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761467.948818:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493949056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761467.948839:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761467.948843:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761467.948847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493949056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761467.948866:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761467.948884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761467.948893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761467/real 1789761467] req@ffff8eb64514b100 x1876699493949184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761467.948917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493949184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761467.948923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761467.948925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761467.948928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949184:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761467.948932:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949312:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761467.948948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761467.948955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761467/real 1789761467] req@ffff8eb64514bb80 x1876699493949312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761467.948963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493949312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761467.948968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761467.948970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761467.948972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949312:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761467.948975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493949440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761467.948986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761467.948991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761467/real 1789761467] req@ffff8eb645148000 x1876699493949440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761467.948999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493949440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761467.949003:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761467.949005:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761467.949007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493949440:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761468.652538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761469.676554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761470.700577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761471.724487:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761472.748564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761473.068492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761473.068504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761473.068508:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761473.068518:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1477 00000100:00080000:2.0:1789761473.068530:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761473.068537:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761473.068588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493949568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761473.068617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761473.068622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761473.068624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761473.068629:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1477 00000100:00080000:2.0:1789761473.068635:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761473.068640:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761473.068650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761473.068653:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493949568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761473.068668:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493949696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761473.068683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761473.068687:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761473.068690:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761473.068694:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1477 00000100:00080000:2.0:1789761473.068700:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761473.068705:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761473.068712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761473.068726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493949824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761473.068739:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949696:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789761473.068740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761473.068748:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761473.068751:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761473.068755:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1477 00000100:00080000:2.0:1789761473.068758:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761473.068762:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761473.068766:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761473.068776:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949824:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761473.068780:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493949952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761473.068804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493949952:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761473.772534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761474.796525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761475.820538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761476.140621:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761476.140657:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761476.140673:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761476.140711:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761476.140729:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761476.140741:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761476.140758:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761473/real 1789761476] req@ffff8eb650f81180 x1876699493949824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761476.140783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493949824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761476.140796:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761476.140801:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761476.140805:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949824:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761476.140812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761473/real 1789761476] req@ffff8eb650f81f80 x1876699493949696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761476.140825:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493949696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761476.140832:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761476.140851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761476.140855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493949696:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761476.140861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761473/real 1789761476] req@ffff8eb650f83b80 x1876699493949568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761476.140872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493949568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761476.140879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761476.140883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761476.140887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493949568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761476.140911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761473/real 1789761476] req@ffff8eb650f83800 x1876699493949952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761476.140922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493949952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761476.140929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761476.140932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761476.140935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493949952:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761476.844539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761477.868514:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761478.188503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761478.188514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761478.188518:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761478.188535:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1482 00000100:00080000:1.0:1789761478.188544:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761478.188550:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761478.188577:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493950080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761478.188601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761478.188605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761478.188607:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761478.188612:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1482 00000100:00080000:1.0:1789761478.188616:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761478.188621:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761478.188630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761478.188644:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493950208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761478.188646:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493950080:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761478.188655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761478.188658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761478.188660:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761478.188663:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1482 00000100:00080000:1.0:1789761478.188667:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761478.188671:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761478.188675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761478.188684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493950336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761478.188694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761478.188697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761478.188698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761478.188702:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1482 00000800:00000100:0.0:1789761478.188704:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761478.188705:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761478.188709:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761478.188714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761478.188723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493950464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761478.188747:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761478/real 1789761478] req@ffff8eb65016ed80 x1876699493950080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761478.188765:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493950080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761478.188775:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761478.188780:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761478.188783:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493950080:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761478.188809:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761478.188860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761478.188874:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761478/real 1789761478] req@ffff8eb65016f480 x1876699493950208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761478.188887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493950208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761478.188896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761478.188899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761478.188914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950208:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761478.188922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950336:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761478.188945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761478.188957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761478/real 1789761478] req@ffff8eb65016dc00 x1876699493950336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761478.188969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493950336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761478.188977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761478.188981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761478.188984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950336:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761478.188991:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493950464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761478.189011:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761478.189023:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761478/real 1789761478] req@ffff8eb65016df80 x1876699493950464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761478.189035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493950464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761478.189049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761478.189052:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761478.189055:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493950464:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761478.892527:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761479.916541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761480.940551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761481.964498:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761482.988526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761483.308500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761483.308508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761483.308512:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761483.308521:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1487 00000100:00080000:3.0:1789761483.308530:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761483.308536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761483.308577:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493950592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761483.308602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761483.308606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761483.308608:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761483.308613:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1487 00000100:00080000:3.0:1789761483.308617:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761483.308622:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761483.308630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761483.308644:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493950592:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761483.308644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493950720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761483.308660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761483.308663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761483.308664:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761483.308667:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1487 00000100:00080000:3.0:1789761483.308670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761483.308672:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761483.308675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761483.308685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493950848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761483.308692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761483.308695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761483.308696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761483.308699:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1487 00000100:00080000:3.0:1789761483.308704:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761483.308706:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761483.308711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761483.308716:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761483.308729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493950976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761483.308758:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761483/real 1789761483] req@ffff8eb64514b800 x1876699493950592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.308776:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493950592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.308804:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.308809:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761483.308813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493950592:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761483.308835:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761483.308857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761483.308868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761483/real 1789761483] req@ffff8eb64514a300 x1876699493950720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.308879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493950720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.308887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.308891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761483.308895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761483.308901:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950848:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761483.308920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761483.308931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761483/real 1789761483] req@ffff8eb645148e00 x1876699493950848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.308943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493950848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.308951:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.308955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761483.308958:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493950848:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761483.308964:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493950976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761483.308982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761483.308992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761483/real 1789761483] req@ffff8eb645148380 x1876699493950976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761483.309005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493950976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761483.309014:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761483.309017:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761483.309021:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493950976:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761484.012531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761485.036546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761486.060536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761487.084511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761488.108550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761488.428493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761488.428507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761488.428510:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761488.428518:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1492 00000100:00080000:1.0:1789761488.428527:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761488.428534:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761488.428570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493951104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761488.428595:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761488.428597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761488.428599:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761488.428602:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1492 00000100:00080000:1.0:1789761488.428605:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761488.428608:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761488.428616:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761488.428627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493951232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761488.428634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493951104:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761488.428638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761488.428642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761488.428643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761488.428647:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1492 00000100:00080000:1.0:1789761488.428649:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761488.428652:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761488.428656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761488.428670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493951360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761488.428688:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761488.428693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789761488.428698:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761488.428698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761488.428708:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1492 00000100:00080000:1.0:1789761488.428713:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761488.428718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761488.428724:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761488.428737:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493951488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761488.428756:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761488/real 1789761488] req@ffff8eb65016c700 x1876699493951104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761488.428780:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493951104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761488.428789:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761488.428792:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761488.428795:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493951104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761488.428814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761488.428832:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761488.428841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761488/real 1789761488] req@ffff8eb65016ce00 x1876699493951232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761488.428850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493951232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761488.428855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761488.428858:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761488.428860:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761488.428864:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951360:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761488.428876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761488.428882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761488/real 1789761488] req@ffff8eb65016e300 x1876699493951360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761488.428894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493951360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761488.428899:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761488.428901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761488.428903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951360:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761488.428906:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493951488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761488.428916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761488.428922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761488/real 1789761488] req@ffff8eb65016fb80 x1876699493951488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761488.428929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493951488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761488.428934:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761488.428936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761488.428938:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493951488:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761489.132501:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761490.156557:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761491.180521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761492.204538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761493.228506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761493.548488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761493.548497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761493.548500:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761493.548508:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1498 00000100:00080000:3.0:1789761493.548517:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761493.548523:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761493.548551:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493951616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761493.548571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761493.548574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761493.548575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761493.548578:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1498 00000100:00080000:3.0:1789761493.548581:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761493.548584:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761493.548592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761493.548601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493951744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761493.548609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761493.548613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761493.548615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761493.548616:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493951616:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761493.548619:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1498 00000100:00080000:3.0:1789761493.548625:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761493.548630:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761493.548635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761493.548648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493951872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761493.548660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761493.548664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761493.548666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761493.548670:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1498 00000100:00080000:3.0:1789761493.548677:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761493.548680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761493.548684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761493.548689:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761493.548698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493952000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761493.548747:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761493/real 1789761493] req@ffff8eb645149c00 x1876699493951616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761493.548762:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493951616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761493.548770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761493.548773:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761493.548776:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493951616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761493.548795:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761493.548812:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761493.548821:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761493/real 1789761493] req@ffff8eb645149500 x1876699493951744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761493.548830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493951744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761493.548835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761493.548838:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761493.548840:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951744:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761493.548844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951872:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761493.548855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761493.548861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761493/real 1789761493] req@ffff8eb645148a80 x1876699493951872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761493.548868:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493951872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761493.548873:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761493.548875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761493.548877:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493951872:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761493.548881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493952000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761493.548892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761493.548897:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761493/real 1789761493] req@ffff8eb64514a680 x1876699493952000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761493.548905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493952000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761493.548910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761493.548911:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761493.548913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493952000:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761494.252520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761495.276548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761496.300506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761497.324536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761498.348537:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761498.668509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761498.668517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761498.668521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761498.668529:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1503 00000100:00080000:1.0:1789761498.668539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761498.668545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761498.668574:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493952128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761498.668596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761498.668599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761498.668600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761498.668603:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1503 00000100:00080000:1.0:1789761498.668605:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761498.668608:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761498.668615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761498.668624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493952256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761498.668632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761498.668635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761498.668636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761498.668638:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1503 00000100:00080000:1.0:1789761498.668642:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761498.668646:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493952128:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761498.668649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761498.668660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761498.668689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493952384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761498.668703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761498.668707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761498.668709:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761498.668714:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1503 00000100:00080000:1.0:1789761498.668719:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761498.668724:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761498.668729:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761498.668733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761498.668751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493952512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761498.668776:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761498/real 1789761498] req@ffff8eb65016d500 x1876699493952128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761498.668799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493952128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761498.668811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761498.668815:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761498.668820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493952128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761498.668828:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761498.668857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761498.668867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761498/real 1789761498] req@ffff8eb65016c380 x1876699493952256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761498.668876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493952256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761498.668881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761498.668884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761498.668886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761498.668903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952384:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761498.668918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761498.668924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761498/real 1789761498] req@ffff8eb65016f800 x1876699493952384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761498.668932:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493952384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761498.668937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761498.668939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761498.668941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952384:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761498.668945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493952512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761498.668955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761498.668961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761498/real 1789761498] req@ffff8eb65016e680 x1876699493952512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761498.668972:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493952512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761498.668977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761498.668979:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761498.668981:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493952512:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761499.372512:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761500.396544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761501.420541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761502.444525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761503.468547:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761503.788512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761503.788521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761503.788525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761503.788534:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1508 00000100:00080000:3.0:1789761503.788544:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761503.788552:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761503.788591:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493952640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761503.788615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761503.788622:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761503.788623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761503.788626:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1508 00000100:00080000:3.0:1789761503.788641:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761503.788646:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761503.788654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761503.788666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493952768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761503.788674:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493952640:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761503.788677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761503.788684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761503.788686:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761503.788689:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1508 00000100:00080000:3.0:1789761503.788692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761503.788697:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761503.788702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761503.788715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493952896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761503.788726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761503.788730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761503.788732:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761503.788736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1508 00000100:00080000:3.0:1789761503.788742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761503.788744:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761503.788750:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761503.788758:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761503.788773:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493953024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761503.788781:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761503/real 1789761503] req@ffff8eb64514a680 x1876699493952640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761503.788799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493952640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761503.788807:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761503.788810:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761503.788814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493952640:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761503.788835:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761503.788850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761503.788858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761503/real 1789761503] req@ffff8eb645148a80 x1876699493952768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761503.788866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493952768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761503.788876:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761503.788878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761503.788880:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952768:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761503.788884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952896:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761503.788895:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761503.788900:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761503/real 1789761503] req@ffff8eb645149500 x1876699493952896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761503.788907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493952896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761503.788912:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761503.788914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761503.788916:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493952896:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761503.788919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493953024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761503.788928:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761503.788934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761503/real 1789761503] req@ffff8eb645149c00 x1876699493953024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761503.788941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493953024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761503.788946:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761503.788947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761503.788950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493953024:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761504.492528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761505.516567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761506.540512:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761507.564553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761508.588517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761508.908468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761508.908476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761508.908479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761508.908487:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1513 00000100:00080000:3.0:1789761508.908495:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761508.908500:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761508.908531:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493953152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761508.908564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761508.908568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761508.908570:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761508.908575:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1513 00000100:00080000:3.0:1789761508.908580:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761508.908585:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761508.908605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761508.908607:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493953152:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761508.908623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493953280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761508.908632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761508.908635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761508.908637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761508.908639:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1513 00000100:00080000:3.0:1789761508.908643:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761508.908646:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761508.908650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761508.908659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493953408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761508.908664:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761508.908675:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761508.908678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761508.908679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761508.908683:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1513 00000100:00080000:3.0:1789761508.908686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761508.908689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761508.908692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761508.908702:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761508/real 1789761508] req@ffff8eb645148380 x1876699493953152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761508.908703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493953536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761508.908743:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493953152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761508.908752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761508.908755:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761508.908758:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493953152:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761508.908776:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761508.908791:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761508.908800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761508/real 1789761508] req@ffff8eb645148e00 x1876699493953280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761508.908809:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493953280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761508.908814:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761508.908816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761508.908818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953280:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761508.908823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953408:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761508.908834:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761508.908840:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761508/real 1789761508] req@ffff8eb64514a300 x1876699493953408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761508.908847:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493953408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761508.908852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761508.908854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761508.908856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953408:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761508.908859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493953536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761508.908870:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761508.908875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761508/real 1789761508] req@ffff8eb64514b800 x1876699493953536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761508.908882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493953536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761508.908887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761508.908889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761508.908890:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493953536:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761509.612538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761510.636504:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761511.660534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761512.684502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761513.708524:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761514.029475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761514.029483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761514.029486:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761514.029495:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1518 00000100:00080000:2.0:1789761514.029504:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761514.029509:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761514.029541:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493953664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761514.029574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761514.029577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761514.029579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761514.029582:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1518 00000100:00080000:2.0:1789761514.029585:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761514.029588:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761514.029597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761514.029612:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493953792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761514.029616:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493953664:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761514.029633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761514.029637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761514.029640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761514.029645:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1518 00000100:00080000:2.0:1789761514.029650:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761514.029654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761514.029659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761514.029670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493953920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761514.029677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761514.029686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761514.029690:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761514.029691:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761514.029695:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1518 00000100:00080000:2.0:1789761514.029698:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761514.029701:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761514.029705:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761514.029715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761514/real 1789761514] req@ffff8eb650f80a80 x1876699493953664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761514.029716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493954048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761514.029760:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493953664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761514.029769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761514.029772:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761514.029775:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493953664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761514.029793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761514.029807:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761514.029815:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761514/real 1789761514] req@ffff8eb650f80000 x1876699493953792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761514.029824:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493953792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761514.029830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761514.029832:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761514.029834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953792:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761514.029837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953920:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761514.029848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761514.029854:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761514/real 1789761514] req@ffff8eb650f80380 x1876699493953920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761514.029861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493953920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761514.029871:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761514.029873:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761514.029875:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493953920:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761514.029878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493954048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761514.029888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761514.029894:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761514/real 1789761514] req@ffff8eb650f81500 x1876699493954048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761514.029901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493954048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761514.029906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761514.029908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761514.029910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493954048:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761514.732581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761515.756514:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761516.780574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761517.804540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761518.828566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761519.148476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761519.148487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761519.148491:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761519.148501:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1523 00000100:00080000:2.0:1789761519.148511:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761519.148533:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761519.148568:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493954176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761519.148596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761519.148600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761519.148602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761519.148607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1523 00000100:00080000:2.0:1789761519.148611:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761519.148616:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761519.148625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761519.148639:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493954176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761519.148641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493954304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761519.148658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761519.148661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761519.148663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761519.148667:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1523 00000100:00080000:2.0:1789761519.148671:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761519.148675:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761519.148681:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761519.148694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493954432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761519.148699:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761519.148711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761519.148715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761519.148717:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761519.148722:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1523 00000100:00080000:2.0:1789761519.148725:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761519.148729:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761519.148734:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761519.148743:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761519/real 1789761519] req@ffff8eb650f81500 x1876699493954176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761519.148749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493954560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761519.148766:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493954176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761519.148775:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761519.148779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761519.148783:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493954176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761519.148819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761519.148845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761519.148855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761519/real 1789761519] req@ffff8eb650f80380 x1876699493954304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761519.148864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493954304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761519.148870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761519.148872:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761519.148874:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954304:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761519.148878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954432:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761519.148891:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761519.148898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761519/real 1789761519] req@ffff8eb650f80000 x1876699493954432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761519.148906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493954432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761519.148911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761519.148913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761519.148915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954432:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761519.148919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493954560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761519.148930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761519.148937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761519/real 1789761519] req@ffff8eb650f80a80 x1876699493954560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761519.148945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493954560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761519.148950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761519.148952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761519.148954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493954560:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761519.852523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761520.876566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761521.900564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761522.924623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761523.948599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761524.268534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761524.268545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761524.268549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761524.268560:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1528 00000100:00080000:3.0:1789761524.268573:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761524.268580:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761524.268629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493954688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761524.268660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761524.268663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761524.268665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761524.268668:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1528 00000100:00080000:3.0:1789761524.268673:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761524.268677:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761524.268693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761524.268708:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493954688:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761524.268710:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493954816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761524.268724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761524.268727:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761524.268728:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761524.268731:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1528 00000100:00080000:3.0:1789761524.268734:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761524.268737:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761524.268741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761524.268750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493954944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761524.268758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761524.268761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761524.268762:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761524.268765:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1528 00000100:00080000:3.0:1789761524.268768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761524.268770:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761524.268773:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761524.268780:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493955072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761524.268797:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761524.268849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761524/real 1789761524] req@ffff8eb645148000 x1876699493954688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761524.268865:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493954688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761524.268874:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761524.268877:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761524.268881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493954688:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761524.268905:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761524.268923:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761524.268932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761524/real 1789761524] req@ffff8eb64514bb80 x1876699493954816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761524.268945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493954816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761524.268952:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761524.268954:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761524.268956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954816:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761524.268960:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954944:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761524.268971:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761524.268977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761524/real 1789761524] req@ffff8eb64514b100 x1876699493954944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761524.268985:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493954944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761524.268990:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761524.268992:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761524.268994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493954944:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761524.268997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493955072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761524.269008:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761524.269014:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761524/real 1789761524] req@ffff8eb645149880 x1876699493955072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761524.269021:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493955072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761524.269031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761524.269033:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761524.269035:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493955072:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761524.972529:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761525.996540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761527.020582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761528.044591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761529.068577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761529.388535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761529.388542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761529.388546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761529.388554:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1533 00000100:00080000:1.0:1789761529.388563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761529.388569:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761529.388601:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493955200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761529.388625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761529.388628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761529.388630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761529.388633:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1533 00000100:00080000:1.0:1789761529.388635:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761529.388638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761529.388646:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761529.388656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493955328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761529.388669:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761529.388674:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493955200:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761529.388674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761529.388678:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761529.388685:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1533 00000100:00080000:1.0:1789761529.388687:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761529.388690:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761529.388694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761529.388705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493955456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761529.388712:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761529.388715:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761529.388717:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761529.388720:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1533 00000100:00080000:1.0:1789761529.388732:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761529.388735:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761529.388739:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761529.388750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493955584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761529.388752:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761529.388804:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761529/real 1789761529] req@ffff8eb65016d880 x1876699493955200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761529.388820:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493955200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761529.388829:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761529.388833:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761529.388836:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493955200:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761529.388857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761529.388876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761529.388885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761529/real 1789761529] req@ffff8eb65016ea00 x1876699493955328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761529.388894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493955328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761529.388905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761529.388907:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761529.388910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955328:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761529.388914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955456:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761529.388925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761529.388932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761529/real 1789761529] req@ffff8eb65016e680 x1876699493955456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761529.388939:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493955456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761529.388945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761529.388947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761529.388949:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955456:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761529.388953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493955584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761529.388963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761529.388969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761529/real 1789761529] req@ffff8eb65016f800 x1876699493955584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761529.388976:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493955584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761529.388981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761529.388983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761529.388986:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493955584:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761530.092628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761531.116606:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761532.140546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761533.164597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761534.188561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761534.508513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761534.508524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761534.508529:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761534.508540:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1539 00000100:00080000:2.0:1789761534.508553:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761534.508562:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761534.508612:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493955712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761534.508645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761534.508648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761534.508650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761534.508654:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1539 00000100:00080000:2.0:1789761534.508658:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761534.508662:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761534.508680:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761534.508694:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493955712:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761534.508694:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493955840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761534.508711:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761534.508714:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761534.508716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761534.508719:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1539 00000100:00080000:2.0:1789761534.508721:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761534.508724:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761534.508727:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761534.508736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493955968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761534.508744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761534.508747:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761534.508749:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761534.508752:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1539 00000100:00080000:2.0:1789761534.508755:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761534.508758:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761534.508761:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761534.508768:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493956096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761534.508780:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761534.508823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761534/real 1789761534] req@ffff8eb650f83800 x1876699493955712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761534.508838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493955712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761534.508847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761534.508850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761534.508853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493955712:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761534.508878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761534.508895:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761534.508903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761534/real 1789761534] req@ffff8eb650f83b80 x1876699493955840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761534.508912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493955840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761534.508918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761534.508920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761534.508922:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955840:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761534.508928:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955968:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761534.508940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761534.508946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761534/real 1789761534] req@ffff8eb650f81f80 x1876699493955968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761534.508954:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493955968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761534.508960:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761534.508961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761534.508963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493955968:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761534.508966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493956096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761534.508977:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761534.508983:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761534/real 1789761534] req@ffff8eb650f81180 x1876699493956096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761534.508991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493956096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761534.508996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761534.508998:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761534.508999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493956096:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761535.212566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761536.236565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761537.260528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761538.284570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761539.308550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761539.628531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761539.628546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761539.628551:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761539.628565:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1544 00000100:00080000:1.0:1789761539.628587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761539.628597:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761539.628646:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493956224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761539.628681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761539.628685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761539.628687:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761539.628693:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1544 00000100:00080000:1.0:1789761539.628698:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761539.628704:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761539.628716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761539.628719:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493956224:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761539.628734:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493956352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761539.628747:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761539.628752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761539.628754:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761539.628759:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1544 00000100:00080000:1.0:1789761539.628764:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761539.628768:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761539.628775:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761539.628789:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493956480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761539.628802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761539.628806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761539.628808:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761539.628813:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1544 00000100:00080000:1.0:1789761539.628819:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761539.628824:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761539.628830:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761539.628846:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493956608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761539.628851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761539.628886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956480:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761539.628909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493956608:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761540.332575:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761541.356505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761542.380571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761542.700701:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761542.700759:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761542.700779:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761542.700818:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761542.700826:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761542.700836:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761542.700873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761539/real 1789761542] req@ffff8eb65016e300 x1876699493956608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761542.700900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493956608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761542.700915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761542.700920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761542.700926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493956608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761542.700936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761539/real 1789761542] req@ffff8eb65016d500 x1876699493956352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761542.700949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493956352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761542.700958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761542.700962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761542.700966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761542.700973:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761539/real 1789761542] req@ffff8eb65016fb80 x1876699493956480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761542.700987:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493956480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761542.700995:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761542.700999:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761542.701002:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956480:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761542.701009:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761539/real 1789761542] req@ffff8eb65016c380 x1876699493956224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761542.701021:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493956224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761542.701029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761542.701033:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761542.701037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493956224:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789761543.404587:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761544.428613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761544.748489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761544.748503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761544.748509:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761544.748523:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1549 00000100:00080000:2.0:1789761544.748537:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761544.748546:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761544.748593:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493956736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761544.748626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761544.748631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761544.748633:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761544.748637:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1549 00000100:00080000:2.0:1789761544.748642:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761544.748647:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761544.748658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761544.748673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493956864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761544.748685:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493956736:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761544.748689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761544.748693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761544.748698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761544.748705:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1549 00000100:00080000:2.0:1789761544.748709:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761544.748713:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761544.748718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761544.748733:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493956992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761544.748744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761544.748749:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761544.748751:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761544.748756:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1549 00000100:00080000:2.0:1789761544.748760:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761544.748765:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761544.748771:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761544.748786:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493957120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761544.748788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761544.748847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761544/real 1789761544] req@ffff8eb650f83480 x1876699493956736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761544.748872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493956736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761544.748885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761544.748891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761544.748895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493956736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761544.748928:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761544.748959:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761544.748975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761544/real 1789761544] req@ffff8eb650f80e00 x1876699493956864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761544.748988:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493956864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761544.748997:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761544.749000:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761544.749004:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956864:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761544.749011:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956992:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761544.749035:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761544.749047:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761544/real 1789761544] req@ffff8eb650f80700 x1876699493956992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761544.749058:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493956992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761544.749067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761544.749070:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761544.749073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493956992:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761544.749079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493957120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761544.749099:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761544.749112:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761544/real 1789761544] req@ffff8eb650f82a00 x1876699493957120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761544.749124:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493957120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761544.749132:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761544.749136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761544.749139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493957120:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761545.452521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761546.476549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761547.500596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761548.524569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761549.548544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761549.868490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761549.868501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761549.868505:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761549.868516:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1554 00000100:00080000:1.0:1789761549.868528:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761549.868535:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761549.868585:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493957248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761549.868607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761549.868610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761549.868612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761549.868615:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1554 00000100:00080000:1.0:1789761549.868618:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761549.868621:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761549.868631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761549.868642:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493957376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761549.868650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761549.868654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761549.868655:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761549.868658:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1554 00000100:00080000:1.0:1789761549.868662:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761549.868666:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761549.868673:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493957248:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789761549.868674:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761549.868698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493957504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761549.868711:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761549.868713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761549.868715:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761549.868720:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1554 00000100:00080000:1.0:1789761549.868724:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761549.868727:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761549.868732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761549.868740:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493957632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761549.868756:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761549.868801:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761549/real 1789761549] req@ffff8eb65016ce00 x1876699493957248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761549.868816:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493957248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761549.868825:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761549.868828:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761549.868831:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493957248:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761549.868837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761549.868852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761549.868860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761549/real 1789761549] req@ffff8eb65016c700 x1876699493957376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761549.868879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493957376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761549.868891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761549.868894:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761549.868898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957376:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761549.868924:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957504:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761549.868948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761549.868961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761549/real 1789761549] req@ffff8eb65016df80 x1876699493957504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761549.868973:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493957504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761549.868982:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761549.868986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761549.868989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957504:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761549.868996:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493957632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761549.869019:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761549.869030:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761549/real 1789761549] req@ffff8eb65016dc00 x1876699493957632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761549.869043:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493957632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761549.869059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761549.869063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761549.869066:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493957632:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761550.572542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761551.596538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761552.620546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761553.644625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761554.668535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761554.988493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761554.988503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761554.988508:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761554.988518:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1559 00000100:00080000:3.0:1789761554.988541:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761554.988551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761554.988600:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493957760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761554.988624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761554.988627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761554.988629:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761554.988633:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1559 00000100:00080000:3.0:1789761554.988636:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761554.988639:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761554.988650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761554.988659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493957888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761554.988667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761554.988670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761554.988672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761554.988677:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493957760:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761554.988678:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1559 00000100:00080000:3.0:1789761554.988681:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761554.988687:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761554.988692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761554.988701:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493958016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761554.988709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761554.988713:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761554.988715:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761554.988719:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1559 00000100:00080000:3.0:1789761554.988724:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761554.988729:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761554.988735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761554.988747:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493958144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761554.988757:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761554.988799:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761554/real 1789761554] req@ffff8eb64514ad80 x1876699493957760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761554.988815:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493957760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761554.988824:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761554.988827:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761554.988830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493957760:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761554.988859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761554.988884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761554.988897:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761554/real 1789761554] req@ffff8eb64514b480 x1876699493957888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761554.988906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493957888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761554.988918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761554.988922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761554.988926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493957888:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761554.988933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958016:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761554.988956:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761554.988968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761554/real 1789761554] req@ffff8eb645148700 x1876699493958016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761554.988982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493958016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761554.988992:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761554.988995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761554.988997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958016:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761554.989002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493958144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761554.989020:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761554.989028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761554/real 1789761554] req@ffff8eb645149180 x1876699493958144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761554.989036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493958144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761554.989041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761554.989043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761554.989046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493958144:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761555.692585:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761556.716533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761557.740540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761558.764545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761559.788558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761560.108517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761560.108536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761560.108540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761560.108551:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1564 00000100:00080000:1.0:1789761560.108566:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761560.108575:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761560.108626:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493958272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761560.108654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761560.108657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761560.108661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761560.108666:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1564 00000100:00080000:1.0:1789761560.108669:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761560.108673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761560.108683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761560.108696:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493958400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761560.108709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493958272:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761560.108709:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761560.108718:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761560.108722:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761560.108728:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1564 00000100:00080000:1.0:1789761560.108733:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761560.108738:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761560.108745:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761560.108757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493958528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761560.108765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761560.108768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761560.108769:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761560.108772:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1564 00000100:00080000:1.0:1789761560.108777:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761560.108780:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761560.108784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761560.108793:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493958656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761560.108795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761560.108858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761560/real 1789761560] req@ffff8eb65016f480 x1876699493958272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761560.108875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493958272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761560.108885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761560.108888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761560.108891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493958272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761560.108915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761560.108934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761560.108943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761560/real 1789761560] req@ffff8eb65016ed80 x1876699493958400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761560.108951:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493958400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761560.108957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761560.108960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761560.108962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958400:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761560.108968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958528:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761560.108980:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761560.108986:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761560/real 1789761560] req@ffff8eb65016d180 x1876699493958528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761560.108994:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493958528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761560.109000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761560.109001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761560.109003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958528:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761560.109007:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493958656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761560.109018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761560.109024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761560/real 1789761560] req@ffff8eb65016c000 x1876699493958656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761560.109031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493958656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761560.109036:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761560.109038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761560.109040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493958656:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761560.812566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761561.836553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761562.860505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761563.884533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761564.908519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761565.228504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761565.228513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761565.228516:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761565.228523:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1569 00000100:00080000:3.0:1789761565.228532:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761565.228537:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761565.228567:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493958784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761565.228588:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761565.228591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761565.228592:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761565.228595:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1569 00000100:00080000:3.0:1789761565.228597:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761565.228600:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761565.228608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761565.228616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493958912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761565.228624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761565.228627:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493958784:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761565.228629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761565.228633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761565.228639:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1569 00000100:00080000:3.0:1789761565.228640:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761565.228642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761565.228645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761565.228654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493959040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761565.228661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761565.228664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761565.228665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761565.228667:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1569 00000100:00080000:3.0:1789761565.228670:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761565.228672:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761565.228675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761565.228684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493959168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761565.228691:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761565.228734:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761565/real 1789761565] req@ffff8eb645149180 x1876699493958784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761565.228754:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493958784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761565.228765:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761565.228769:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761565.228774:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493958784:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761565.228797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761565.228822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761565.228836:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761565/real 1789761565] req@ffff8eb645148700 x1876699493958912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761565.228850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493958912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761565.228859:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761565.228863:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761565.228867:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493958912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761565.228874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959040:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761565.228896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761565.228907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761565/real 1789761565] req@ffff8eb64514b480 x1876699493959040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761565.228927:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493959040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761565.228937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761565.228940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761565.228943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959040:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761565.228950:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493959168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761565.228969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761565.228977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761565/real 1789761565] req@ffff8eb64514ad80 x1876699493959168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761565.228985:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493959168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761565.228990:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761565.228992:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761565.228994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493959168:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761565.932603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761566.956529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761567.980567:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761569.004520:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761570.028527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761570.348525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761570.348540:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761570.348545:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761570.348560:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1574 00000100:00080000:3.0:1789761570.348575:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761570.348584:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761570.348641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493959296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761570.348666:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761570.348669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761570.348670:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761570.348674:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1574 00000100:00080000:3.0:1789761570.348677:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761570.348680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761570.348691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761570.348701:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493959424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761570.348709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493959296:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761570.348717:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761570.348722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761570.348724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761570.348729:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1574 00000100:00080000:3.0:1789761570.348733:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761570.348738:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761570.348744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761570.348758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493959552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761570.348769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761570.348773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761570.348774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761570.348777:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1574 00000100:00080000:3.0:1789761570.348781:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761570.348786:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761570.348789:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761570.348797:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761570.348810:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493959680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761570.348845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761570/real 1789761570] req@ffff8eb645149880 x1876699493959296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761570.348870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493959296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761570.348879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761570.348883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761570.348886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493959296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761570.348911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761570.348931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761570.348941:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761570/real 1789761570] req@ffff8eb64514b100 x1876699493959424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761570.348955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493959424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761570.348961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761570.348964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761570.348966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959424:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761570.348970:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959552:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761570.348984:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761570.348991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761570/real 1789761570] req@ffff8eb64514bb80 x1876699493959552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761570.348999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493959552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761570.349009:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761570.349012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761570.349014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959552:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761570.349017:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493959680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761570.349033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761570.349044:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761570/real 1789761570] req@ffff8eb645148000 x1876699493959680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761570.349057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493959680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761570.349066:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761570.349070:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761570.349074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493959680:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761571.052537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761572.076534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761573.100551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761574.124532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761575.148555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761575.468507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761575.468518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761575.468523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761575.468531:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1579 00000100:00080000:2.0:1789761575.468540:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761575.468546:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761575.468576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493959808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761575.468597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761575.468600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761575.468602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761575.468605:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1579 00000100:00080000:2.0:1789761575.468607:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761575.468609:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761575.468617:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761575.468631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493959936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761575.468639:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493959808:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761575.468643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761575.468647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761575.468648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761575.468652:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1579 00000100:00080000:2.0:1789761575.468654:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761575.468657:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761575.468661:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761575.468669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493960064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761575.468676:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761575.468679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761575.468680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761575.468683:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1579 00000100:00080000:2.0:1789761575.468692:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761575.468696:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761575.468700:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789761575.468702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761575.468712:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493960192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761575.468737:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761575/real 1789761575] req@ffff8eb650f81c00 x1876699493959808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761575.468750:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493959808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761575.468762:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761575.468765:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761575.468768:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493959808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761575.468786:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761575.468800:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761575.468808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761575/real 1789761575] req@ffff8eb650f82680 x1876699493959936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761575.468817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493959936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761575.468822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761575.468824:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761575.468826:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493959936:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761575.468831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960064:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761575.468849:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761575.468860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761575/real 1789761575] req@ffff8eb650f82d80 x1876699493960064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761575.468887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493960064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761575.468896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761575.468899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761575.468903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960064:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761575.468908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493960192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761575.468924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761575.468932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761575/real 1789761575] req@ffff8eb650f81880 x1876699493960192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761575.468940:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493960192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761575.468945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761575.468947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761575.468950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493960192:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761576.172539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761577.196537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761578.220531:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761579.244552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761580.268613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761580.588545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761580.588558:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761580.588562:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761580.588573:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1585 00000100:00080000:2.0:1789761580.588585:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761580.588593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761580.588652:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493960320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761580.588690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761580.588693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761580.588695:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761580.588699:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1585 00000100:00080000:2.0:1789761580.588702:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761580.588705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761580.588716:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761580.588728:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493960448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761580.588739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761580.588741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493960320:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761580.588743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761580.588747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761580.588751:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1585 00000100:00080000:2.0:1789761580.588753:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761580.588755:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761580.588759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761580.588768:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493960576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761580.588775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761580.588778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761580.588780:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761580.588782:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1585 00000100:00080000:2.0:1789761580.588785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761580.588787:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761580.588790:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761580.588798:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493960704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761580.588816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761580.588860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761580/real 1789761580] req@ffff8eb650f81880 x1876699493960320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761580.588875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493960320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761580.588885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761580.588888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761580.588891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493960320:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761580.588914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761580.588930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761580.588940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761580/real 1789761580] req@ffff8eb650f82d80 x1876699493960448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761580.588948:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493960448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761580.588953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761580.588956:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761580.588958:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960448:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761580.588961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960576:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761580.588974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761580.588980:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761580/real 1789761580] req@ffff8eb650f82680 x1876699493960576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761580.588987:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493960576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761580.588993:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761580.588995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761580.588997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960576:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761580.589001:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493960704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761580.589012:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761580.589018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761580/real 1789761580] req@ffff8eb650f81c00 x1876699493960704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761580.589030:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493960704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761580.589035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761580.589037:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761580.589039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493960704:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761581.292603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761582.316567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761583.340625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761584.364576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761585.388551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761585.708487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761585.708499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761585.708503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761585.708515:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1590 00000100:00080000:3.0:1789761585.708533:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761585.708540:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761585.708583:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493960832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761585.708606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761585.708609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761585.708611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761585.708614:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1590 00000100:00080000:3.0:1789761585.708616:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761585.708619:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761585.708630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761585.708639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493960960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761585.708648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761585.708652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761585.708654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761585.708655:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493960832:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761585.708659:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1590 00000100:00080000:3.0:1789761585.708663:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761585.708667:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761585.708671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761585.708680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493961088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761585.708686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761585.708689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761585.708690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761585.708694:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1590 00000100:00080000:3.0:1789761585.708696:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761585.708699:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761585.708701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761585.708708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493961216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761585.708755:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761585.708797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761585/real 1789761585] req@ffff8eb64514b800 x1876699493960832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761585.708812:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493960832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761585.708822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761585.708853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761585.708857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493960832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761585.708885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761585.708908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761585.708945:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761585/real 1789761585] req@ffff8eb64514a300 x1876699493960960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761585.708959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493960960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761585.708965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761585.708967:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761585.708970:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493960960:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761585.708977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961088:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761585.709000:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761585.709012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761585/real 1789761585] req@ffff8eb645148e00 x1876699493961088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761585.709041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493961088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761585.709059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761585.709061:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761585.709064:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961088:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761585.709068:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493961216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761585.709085:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761585.709093:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761585/real 1789761585] req@ffff8eb645148380 x1876699493961216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761585.709101:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493961216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761585.709113:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761585.709117:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761585.709120:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493961216:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761586.412527:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761587.436598:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761588.460518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761589.484625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761590.508549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761590.828496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761590.828511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761590.828517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761590.828544:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1595 00000100:00080000:3.0:1789761590.828558:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761590.828568:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761590.828617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493961344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761590.828651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761590.828655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761590.828658:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761590.828664:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1595 00000100:00080000:3.0:1789761590.828670:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761590.828675:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761590.828688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761590.828689:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493961344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761590.828703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493961472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761590.828713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761590.828716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761590.828718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761590.828721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1595 00000100:00080000:3.0:1789761590.828724:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761590.828726:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761590.828731:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761590.828739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493961600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761590.828747:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761590.828750:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761590.828751:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761590.828754:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1595 00000100:00080000:3.0:1789761590.828758:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761590.828760:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761590.828763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761590.828770:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761590.828773:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493961728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761590.828832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761590/real 1789761590] req@ffff8eb645149c00 x1876699493961344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761590.828855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493961344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761590.828867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761590.828872:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761590.828876:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493961344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761590.828900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761590.828924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761590.828938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761590/real 1789761590] req@ffff8eb645149500 x1876699493961472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761590.828952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493961472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761590.828970:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761590.828973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761590.828975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961472:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761590.828994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961600:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761590.829020:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761590.829033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761590/real 1789761590] req@ffff8eb645148a80 x1876699493961600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761590.829047:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493961600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761590.829056:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761590.829060:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761590.829064:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961600:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761590.829071:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493961728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761590.829096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761590.829110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761590/real 1789761590] req@ffff8eb64514a680 x1876699493961728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761590.829122:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493961728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761590.829131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761590.829135:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761590.829139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493961728:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761591.532567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761592.556618:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761593.580542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761594.604626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761595.628580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761595.948485:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761595.948492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761595.948496:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761595.948505:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1600 00000100:00080000:3.0:1789761595.948517:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761595.948525:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761595.948572:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493961856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761595.948600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761595.948609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761595.948612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761595.948618:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1600 00000100:00080000:3.0:1789761595.948622:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761595.948627:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761595.948638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761595.948653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493961984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761595.948654:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493961856:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761595.948672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761595.948676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761595.948678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761595.948683:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1600 00000100:00080000:3.0:1789761595.948687:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761595.948691:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761595.948697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761595.948707:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493962112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761595.948714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761595.948716:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761595.948719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761595.948722:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761595.948726:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1600 00000100:00080000:3.0:1789761595.948730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761595.948734:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761595.948738:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761595.948746:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493962240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761595.948754:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761595/real 1789761595] req@ffff8eb64514a680 x1876699493961856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761595.948768:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493961856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761595.948776:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761595.948779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761595.948782:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493961856:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761595.948800:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761595.948815:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761595.948823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761595/real 1789761595] req@ffff8eb645148a80 x1876699493961984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761595.948830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493961984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761595.948836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761595.948838:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761595.948840:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493961984:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761595.948847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962112:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761595.948858:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761595.948863:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761595/real 1789761595] req@ffff8eb645149500 x1876699493962112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761595.948871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493962112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761595.948876:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761595.948878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761595.948880:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962112:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761595.948883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493962240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761595.948892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761595.948907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761595/real 1789761595] req@ffff8eb645149c00 x1876699493962240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761595.948919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493962240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761595.948928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761595.948931:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761595.948935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493962240:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761596.652643:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761597.676527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761598.700633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761599.724613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761600.748553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761601.068499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761601.068510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761601.068514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761601.068524:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1605 00000100:00080000:1.0:1789761601.068536:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761601.068543:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761601.068584:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493962368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761601.068607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761601.068610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761601.068611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761601.068615:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1605 00000100:00080000:1.0:1789761601.068617:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761601.068620:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761601.068629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761601.068639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493962496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761601.068649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493962368:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761601.068650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761601.068658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761601.068660:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761601.068664:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1605 00000100:00080000:1.0:1789761601.068666:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761601.068668:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761601.068672:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761601.068683:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493962624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761601.068690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761601.068692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761601.068694:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761601.068697:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1605 00000100:00080000:1.0:1789761601.068712:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761601.068715:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761601.068718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761601.068729:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761601.068729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493962752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761601.068774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761601/real 1789761601] req@ffff8eb65016f100 x1876699493962368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761601.068789:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493962368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761601.068798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761601.068801:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761601.068804:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493962368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761601.068828:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761601.068843:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761601.068852:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761601/real 1789761601] req@ffff8eb65016ca80 x1876699493962496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761601.068860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493962496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761601.068866:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761601.068868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761601.068871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962496:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761601.068874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962624:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761601.068885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761601.068891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761601/real 1789761601] req@ffff8eb65016c000 x1876699493962624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761601.068923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493962624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761601.068932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761601.068936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761601.068940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493962624:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761601.068946:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493962752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761601.068969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761601.068978:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761601/real 1789761601] req@ffff8eb65016d180 x1876699493962752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761601.068986:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493962752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761601.068992:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761601.068994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761601.068996:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493962752:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761601.772524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761602.796522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761603.820602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761604.844564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761605.868508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761606.188523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761606.188534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761606.188539:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761606.188553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1610 00000100:00080000:2.0:1789761606.188568:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761606.188578:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761606.188627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493962880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761606.188655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761606.188659:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761606.188661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761606.188666:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1610 00000100:00080000:2.0:1789761606.188670:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761606.188675:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761606.188688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761606.188709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493962880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761606.188709:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493963008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761606.188726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761606.188729:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761606.188732:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761606.188736:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1610 00000100:00080000:2.0:1789761606.188740:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761606.188745:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761606.188751:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761606.188764:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493963136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761606.188776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761606.188781:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761606.188783:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761606.188787:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1610 00000100:00080000:2.0:1789761606.188792:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761606.188797:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761606.188802:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761606.188813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493963264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761606.188831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761606.188858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963136:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761606.188907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493963264:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761606.892620:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761607.916528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761608.940544:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761609.260760:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789761609.260830:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761609.260866:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761609.260919:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761609.260932:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789761609.260949:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789761609.260971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761606/real 1789761609] req@ffff8eb650f80700 x1876699493963008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.261003:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493963008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.261019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.261026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761609.261032:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761609.261042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761606/real 1789761609] req@ffff8eb650f80e00 x1876699493963136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.261057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493963136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.261067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.261071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761609.261075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963136:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761609.261084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761606/real 1789761609] req@ffff8eb650f83480 x1876699493963264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.261097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493963264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.261111:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.261115:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761609.261119:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493963264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761609.261127:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761606/real 1789761609] req@ffff8eb650f82a00 x1876699493962880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.261141:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493962880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.261148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.261152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761609.261155:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493962880:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761609.964517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761610.988538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761611.308521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761611.308536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761611.308540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761611.308551:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1615 00000100:00080000:2.0:1789761611.308564:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761611.308573:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761611.308616:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493963392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761611.308651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761611.308655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761611.308661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761611.308665:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1615 00000100:00080000:2.0:1789761611.308671:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761611.308677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761611.308689:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761611.308698:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493963392:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761611.308707:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493963520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761611.308732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761611.308740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761611.308743:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761611.308761:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1615 00000100:00080000:2.0:1789761611.308766:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761611.308770:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761611.308777:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761611.308792:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761611.308793:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493963648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761611.308807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761611.308811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761611.308813:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761611.308818:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1615 00000100:00080000:2.0:1789761611.308822:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761611.308839:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761611.308845:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761611.308859:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493963776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761611.308873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761611/real 1789761611] req@ffff8eb650f81180 x1876699493963392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761611.308897:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493963392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761611.308911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761611.308915:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761611.308920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493963392:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761611.308954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761611.308983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761611.308997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761611/real 1789761611] req@ffff8eb650f81f80 x1876699493963520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761611.309011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493963520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761611.309021:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761611.309025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761611.309029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963520:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761611.309037:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963648:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761611.309060:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761611.309072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761611/real 1789761611] req@ffff8eb650f83b80 x1876699493963648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761611.309083:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493963648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761611.309092:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761611.309096:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761611.309099:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493963648:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761611.309104:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493963776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761611.309124:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761611.309135:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761611/real 1789761611] req@ffff8eb650f83800 x1876699493963776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761611.309153:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493963776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761611.309161:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761611.309164:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761611.309167:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493963776:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761612.012566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761613.036529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761614.060760:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761615.084550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761616.108638:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761616.428512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761616.428541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761616.428546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761616.428556:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1620 00000100:00080000:3.0:1789761616.428567:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761616.428581:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761616.428619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493963904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761616.428653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761616.428658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761616.428660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761616.428664:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1620 00000100:00080000:3.0:1789761616.428669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761616.428674:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761616.428684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761616.428703:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493963904:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761616.428703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493964032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761616.428721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761616.428725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761616.428727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761616.428731:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1620 00000100:00080000:3.0:1789761616.428735:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761616.428739:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761616.428743:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761616.428756:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493964160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761616.428760:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761616.428770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761616.428773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761616.428774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761616.428779:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1620 00000100:00080000:3.0:1789761616.428781:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761616.428786:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761616.428809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761616.428816:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761616/real 1789761616] req@ffff8eb645148380 x1876699493963904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761616.428825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493964288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761616.428841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493963904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761616.428855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761616.428860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761616.428865:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493963904:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761616.428891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761616.428917:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761616.428931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761616/real 1789761616] req@ffff8eb645148e00 x1876699493964032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761616.428952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493964032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761616.428961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761616.428965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761616.428968:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964032:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761616.428974:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964160:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761616.428997:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761616.429009:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761616/real 1789761616] req@ffff8eb64514a300 x1876699493964160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761616.429022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493964160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761616.429031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761616.429035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761616.429038:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964160:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761616.429044:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493964288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761616.429068:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761616.429081:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761616/real 1789761616] req@ffff8eb64514b800 x1876699493964288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761616.429095:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493964288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761616.429110:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761616.429114:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761616.429118:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493964288:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761617.132592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761618.156548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761619.180579:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761620.204549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761621.228586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761621.548564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761621.548576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761621.548581:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761621.548594:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1626 00000100:00080000:2.0:1789761621.548607:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761621.548615:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761621.548662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493964416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761621.548693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761621.548697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761621.548700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761621.548704:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1626 00000100:00080000:2.0:1789761621.548707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761621.548711:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761621.548721:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761621.548733:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493964544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761621.548742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761621.548746:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761621.548760:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761621.548765:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493964416:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761621.548765:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1626 00000100:00080000:2.0:1789761621.548769:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761621.548775:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761621.548780:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761621.548791:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493964672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761621.548799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761621.548802:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761621.548804:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761621.548807:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1626 00000100:00080000:2.0:1789761621.548809:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761621.548812:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761621.548815:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761621.548822:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493964800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761621.548856:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761621.548910:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761621/real 1789761621] req@ffff8eb650f80a80 x1876699493964416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761621.548931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493964416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761621.548945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761621.548950:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761621.548954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493964416:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761621.548992:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761621.549020:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761621.549034:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761621/real 1789761621] req@ffff8eb650f80000 x1876699493964544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761621.549045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493964544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761621.549056:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761621.549059:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761621.549061:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964544:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761621.549066:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964672:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761621.549092:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761621.549100:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761621/real 1789761621] req@ffff8eb650f80380 x1876699493964672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761621.549108:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493964672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761621.549114:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761621.549116:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761621.549118:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493964672:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761621.549122:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493964800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761621.549132:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761621.549138:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761621/real 1789761621] req@ffff8eb650f81500 x1876699493964800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761621.549145:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493964800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761621.549151:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761621.549152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761621.549155:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493964800:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761622.252615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761623.276576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761624.300622:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761625.324636:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761626.348551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761626.668492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761626.668500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761626.668504:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761626.668517:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1631 00000100:00080000:3.0:1789761626.668526:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761626.668532:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761626.668564:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493964928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761626.668585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761626.668588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761626.668589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761626.668592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1631 00000100:00080000:3.0:1789761626.668595:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761626.668634:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761626.668643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761626.668649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493964928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761626.668665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493965056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761626.668680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761626.668684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761626.668686:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761626.668691:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1631 00000100:00080000:3.0:1789761626.668696:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761626.668703:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761626.668726:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761626.668744:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493965184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761626.668757:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761626.668758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761626.668768:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761626.668770:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761626.668776:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1631 00000100:00080000:3.0:1789761626.668782:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761626.668787:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761626.668792:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761626.668807:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493965312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761626.668823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761626/real 1789761626] req@ffff8eb645148000 x1876699493964928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761626.668847:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493964928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761626.668857:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761626.668860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761626.668863:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493964928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761626.668914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761626.668936:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761626.668946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761626/real 1789761626] req@ffff8eb64514bb80 x1876699493965056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761626.668955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493965056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761626.668961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761626.668964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761626.668966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965056:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761626.668974:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965184:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761626.668987:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761626.668994:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761626/real 1789761626] req@ffff8eb64514b100 x1876699493965184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761626.669002:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493965184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761626.669008:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761626.669010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761626.669012:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965184:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761626.669016:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493965312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761626.669027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761626.669033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761626/real 1789761626] req@ffff8eb645149880 x1876699493965312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761626.669041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493965312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761626.669046:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761626.669048:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761626.669050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493965312:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761627.372522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761628.396523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761629.420585:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761630.444546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761631.468508:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761631.788463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761631.788471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761631.788475:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761631.788482:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1636 00000100:00080000:3.0:1789761631.788498:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761631.788504:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761631.788533:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493965440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761631.788562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761631.788566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761631.788569:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761631.788574:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1636 00000100:00080000:3.0:1789761631.788579:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761631.788583:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761631.788593:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761631.788605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493965568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761631.788613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761631.788616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761631.788618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761631.788621:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1636 00000100:00080000:3.0:1789761631.788624:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761631.788626:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761631.788630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761631.788636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493965696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761631.788645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761631.788650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761631.788654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761631.788656:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493965440:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761631.788660:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1636 00000100:00080000:3.0:1789761631.788666:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761631.788685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761631.788689:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761631.788698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493965824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761631.788768:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761631.788813:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761631/real 1789761631] req@ffff8eb64514ad80 x1876699493965440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761631.788829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493965440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761631.788839:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761631.788843:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761631.788847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493965440:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761631.788856:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761631.788881:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761631.788895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761631/real 1789761631] req@ffff8eb64514b480 x1876699493965568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761631.788910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493965568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761631.788920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761631.788924:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761631.788928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761631.788958:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965696:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761631.788981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761631.788992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761631/real 1789761631] req@ffff8eb645148700 x1876699493965696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761631.789013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493965696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761631.789023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761631.789026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761631.789030:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493965696:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761631.789037:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493965824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761631.789060:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761631.789073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761631/real 1789761631] req@ffff8eb645149180 x1876699493965824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761631.789086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493965824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761631.789095:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761631.789099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761631.789102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493965824:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761632.492524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761633.516600:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761634.540530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761635.564568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761636.588589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761636.908574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761636.908699:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761636.908704:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761636.908715:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1641 00000100:00080000:2.0:1789761636.908726:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761636.908734:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761636.908775:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493965952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761636.908805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761636.908809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761636.908811:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761636.908817:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1641 00000100:00080000:2.0:1789761636.908823:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761636.908829:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761636.908839:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761636.908853:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493965952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761636.908857:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493966080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761636.908887:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761636.908891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761636.908893:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761636.908898:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1641 00000100:00080000:2.0:1789761636.908903:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761636.908907:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761636.908914:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761636.908938:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493966208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761636.908950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761636.908962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761636.908967:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761636.908969:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761636.908976:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1641 00000100:00080000:2.0:1789761636.908982:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761636.908987:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761636.908994:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761636.909004:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761636/real 1789761636] req@ffff8eb650f81500 x1876699493965952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761636.909013:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493966336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761636.909094:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493965952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761636.909106:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761636.909110:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761636.909113:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493965952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761636.909141:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761636.909161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761636.909171:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761636/real 1789761636] req@ffff8eb650f80380 x1876699493966080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761636.909179:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493966080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761636.909186:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761636.909188:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761636.909191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966080:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761636.909195:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966208:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761636.909207:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761636.909213:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761636/real 1789761636] req@ffff8eb650f80000 x1876699493966208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761636.909220:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493966208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761636.909231:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761636.909233:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761636.909235:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966208:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761636.909239:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493966336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761636.909250:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761636.909256:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761636/real 1789761636] req@ffff8eb650f80a80 x1876699493966336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761636.909263:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493966336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761636.909269:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761636.909270:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761636.909272:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493966336:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761637.612553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761638.636546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761639.660580:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761640.684565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761641.708566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761642.028484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761642.028492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761642.028495:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761642.028504:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1646 00000100:00080000:3.0:1789761642.028520:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761642.028526:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761642.028560:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493966464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761642.028579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761642.028582:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761642.028583:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761642.028586:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1646 00000100:00080000:3.0:1789761642.028589:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761642.028591:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761642.028610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761642.028623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493966592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761642.028633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761642.028634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493966464:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761642.028640:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761642.028644:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761642.028648:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1646 00000100:00080000:3.0:1789761642.028651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761642.028653:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761642.028657:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761642.028666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493966720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761642.028674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761642.028676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761642.028678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761642.028681:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1646 00000100:00080000:3.0:1789761642.028683:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761642.028685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761642.028688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761642.028695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493966848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761642.028699:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761642.028735:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761642/real 1789761642] req@ffff8eb645149180 x1876699493966464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761642.028749:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493966464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761642.028761:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761642.028764:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761642.028767:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493966464:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761642.028785:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761642.028800:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761642.028808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761642/real 1789761642] req@ffff8eb645148700 x1876699493966592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761642.028817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493966592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761642.028822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761642.028824:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761642.028826:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966592:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761642.028830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966720:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761642.028841:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761642.028847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761642/real 1789761642] req@ffff8eb64514b480 x1876699493966720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761642.028855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493966720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761642.028860:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761642.028862:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761642.028864:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493966720:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761642.028868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493966848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761642.028878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761642.028884:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761642/real 1789761642] req@ffff8eb64514ad80 x1876699493966848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761642.028891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493966848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761642.028896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761642.028898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761642.028900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493966848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761642.732550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761643.756533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761644.780528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761645.804557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761646.828517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761647.148484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761647.148495:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761647.148500:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761647.148510:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1651 00000100:00080000:3.0:1789761647.148521:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761647.148529:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761647.148579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493966976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761647.148608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761647.148616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761647.148618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761647.148623:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1651 00000100:00080000:3.0:1789761647.148628:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761647.148633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761647.148643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761647.148652:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493966976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761647.148660:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493967104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761647.148674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761647.148678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761647.148680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761647.148692:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1651 00000100:00080000:3.0:1789761647.148697:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761647.148701:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761647.148707:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761647.148712:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761647.148721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493967232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761647.148733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761647.148737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761647.148739:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761647.148744:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1651 00000100:00100000:0.0:1789761647.148748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761647/real 1789761647] req@ffff8eb645149880 x1876699493966976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761647.148750:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761647.148759:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761647.148766:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761647.148769:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493966976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761647.148778:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761647.148783:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493967360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761647.148785:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761647.148793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493966976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761647.148825:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761647.148842:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761647.148850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761647/real 1789761647] req@ffff8eb64514b100 x1876699493967104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761647.148858:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493967104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761647.148864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761647.148866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761647.148868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761647.148872:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967232:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761647.148883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761647.148889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761647/real 1789761647] req@ffff8eb64514bb80 x1876699493967232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761647.148896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493967232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761647.148901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761647.148903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761647.148905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761647.148909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493967360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761647.148919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761647.148925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761647/real 1789761647] req@ffff8eb645148000 x1876699493967360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761647.148933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493967360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761647.148938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761647.148939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761647.148941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493967360:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761647.852526:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761648.876543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761649.900533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761650.924598:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761651.948538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761652.268529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761652.268543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761652.268550:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761652.268564:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1656 00000100:00080000:0.0:1789761652.268578:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761652.268586:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761652.268631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699493967488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761652.268663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761652.268666:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761652.268668:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761652.268671:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1656 00000100:00080000:0.0:1789761652.268674:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761652.268676:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761652.268686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761652.268697:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699493967616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761652.268706:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761652.268708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761652.268710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761652.268716:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1656 00000100:00080000:0.0:1789761652.268718:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761652.268718:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493967488:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761652.268724:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761652.268732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761652.268739:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699493967744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761652.268746:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761652.268749:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761652.268751:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761652.268753:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1656 00000100:00080000:0.0:1789761652.268756:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761652.268758:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761652.268762:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761652.268769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699493967872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761652.268804:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761652.268855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761652/real 1789761652] req@ffff8eb64304f100 x1876699493967488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761652.268886:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699493967488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761652.268899:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761652.268903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761652.268906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493967488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761652.268930:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761652.268957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761652.268968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761652/real 1789761652] req@ffff8eb64304c700 x1876699493967616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761652.268978:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699493967616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761652.268984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761652.268986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761652.268989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967616:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761652.268992:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967744:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761652.269007:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761652.269014:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761652/real 1789761652] req@ffff8eb64304ea00 x1876699493967744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761652.269026:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699493967744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761652.269042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761652.269046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761652.269049:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493967744:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761652.269056:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493967872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761652.269082:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761652.269091:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761652/real 1789761652] req@ffff8eb64304ca80 x1876699493967872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761652.269099:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699493967872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761652.269105:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761652.269107:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761652.269109:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493967872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761652.972572:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761653.996580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761655.020553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761656.044558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761657.068558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761657.388508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761657.388517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761657.388520:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761657.388531:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1661 00000100:00080000:3.0:1789761657.388544:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761657.388552:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761657.388587:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493968000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761657.388608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761657.388611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761657.388612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761657.388616:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1661 00000100:00080000:3.0:1789761657.388618:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761657.388621:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761657.388629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761657.388640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493968128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761657.388649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761657.388651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761657.388653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761657.388657:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1661 00000100:00080000:3.0:1789761657.388664:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761657.388665:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493968000:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761657.388669:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761657.388686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761657.388694:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493968256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761657.388701:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761657.388703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761657.388704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761657.388707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1661 00000100:00080000:3.0:1789761657.388709:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761657.388712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761657.388714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761657.388722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493968384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761657.388765:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761657.388807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761657/real 1789761657] req@ffff8eb64514b800 x1876699493968000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761657.388823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493968000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761657.388833:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761657.388837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761657.388840:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493968000:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761657.388865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761657.388882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761657.388891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761657/real 1789761657] req@ffff8eb64514a300 x1876699493968128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761657.388899:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493968128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761657.388905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761657.388908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761657.388910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761657.388915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968256:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761657.388926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761657.388933:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761657/real 1789761657] req@ffff8eb645148e00 x1876699493968256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761657.388940:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493968256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761657.388946:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761657.388948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761657.388950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968256:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761657.388954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493968384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761657.388965:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761657.388971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761657/real 1789761657] req@ffff8eb645148380 x1876699493968384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761657.388979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493968384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761657.388985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761657.388987:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761657.388989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493968384:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761658.092589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761659.116610:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761660.140586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761661.164581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761662.188551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761662.508490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761662.508498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761662.508502:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761662.508511:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1667 00000100:00080000:3.0:1789761662.508519:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761662.508525:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761662.508555:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493968512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761662.508578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761662.508580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761662.508582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761662.508585:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1667 00000100:00080000:3.0:1789761662.508587:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761662.508589:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761662.508596:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761662.508604:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493968640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761662.508611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761662.508614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761662.508615:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761662.508618:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1667 00000100:00080000:3.0:1789761662.508620:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761662.508622:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761662.508626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761662.508634:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493968768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761662.508648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761662.508649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493968512:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761662.508651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761662.508654:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761662.508662:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1667 00000100:00080000:3.0:1789761662.508664:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761662.508666:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761662.508669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761662.508677:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493968896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761662.508722:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761662.508765:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761662/real 1789761662] req@ffff8eb645149c00 x1876699493968512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761662.508786:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493968512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761662.508798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761662.508802:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761662.508807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493968512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761662.508815:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761662.508840:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761662.508855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761662/real 1789761662] req@ffff8eb645149500 x1876699493968640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761662.508870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493968640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761662.508886:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761662.508889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761662.508893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761662.508919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968768:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761662.508944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761662.508958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761662/real 1789761662] req@ffff8eb645148a80 x1876699493968768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761662.508971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493968768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761662.508980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761662.508984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761662.508988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493968768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761662.508995:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493968896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761662.509017:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761662.509029:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761662/real 1789761662] req@ffff8eb64514a680 x1876699493968896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761662.509042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493968896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761662.509052:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761662.509055:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761662.509059:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493968896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761663.212519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761664.236614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761665.260585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761666.284569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761667.308572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761667.628543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761667.628559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761667.628565:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761667.628579:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1672 00000100:00080000:0.0:1789761667.628597:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761667.628608:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761667.628661:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699493969024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761667.628697:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761667.628702:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761667.628705:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761667.628709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1672 00000100:00080000:0.0:1789761667.628714:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761667.628718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761667.628729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761667.628731:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493969024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761667.628743:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699493969152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761667.628755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761667.628758:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761667.628760:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761667.628763:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1672 00000100:00080000:0.0:1789761667.628767:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761667.628769:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761667.628773:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761667.628783:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699493969280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761667.628791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761667.628794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761667.628795:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761667.628798:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1672 00000100:00080000:0.0:1789761667.628800:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761667.628803:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761667.628806:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761667.628812:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761667.628816:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699493969408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761667.628862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761667/real 1789761667] req@ffff8eb64304e680 x1876699493969024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761667.628878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699493969024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761667.628887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761667.628891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761667.628894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493969024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761667.628922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761667.628940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761667.628949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761667/real 1789761667] req@ffff8eb64304f480 x1876699493969152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761667.628957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699493969152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761667.628963:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761667.628965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761667.628968:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761667.628971:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969280:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761667.628983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761667.628990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761667/real 1789761667] req@ffff8eb64304d880 x1876699493969280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761667.628998:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699493969280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761667.629003:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761667.629005:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761667.629007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761667.629011:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493969408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761667.629022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761667.629028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761667/real 1789761667] req@ffff8eb64304d500 x1876699493969408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761667.629036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699493969408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761667.629041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761667.629043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761667.629045:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493969408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761668.332573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761669.356519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761670.380585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761671.404569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761672.428553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761672.748545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761672.748557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761672.748561:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761672.748573:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1677 00000100:00080000:2.0:1789761672.748592:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761672.748602:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761672.748653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493969536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761672.748686:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761672.748698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761672.748701:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761672.748708:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1677 00000100:00080000:2.0:1789761672.748712:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761672.748718:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493969536:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761672.748719:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761672.748734:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761672.748750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493969664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761672.748760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761672.748763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761672.748766:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761672.748771:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1677 00000100:00080000:2.0:1789761672.748775:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761672.748780:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761672.748787:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761672.748801:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493969792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761672.748814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761672.748818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761672.748821:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761672.748827:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1677 00000100:00080000:2.0:1789761672.748832:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761672.748838:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761672.748841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969664:192.168.203.160@tcp:38:kworker.0 00000008:00100000:2.0:1789761672.748850:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761672.748868:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493969920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761672.748878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761672.748902:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493969920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761673.452526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761674.476637:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761675.500561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761675.820818:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789761675.820880:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761675.820902:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761675.820957:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761675.820968:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761675.820981:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789761675.821018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761672/real 1789761675] req@ffff8eb650f81180 x1876699493969920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761675.821049:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493969920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761675.821063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761675.821070:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761675.821075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493969920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761675.821084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761672/real 1789761675] req@ffff8eb650f83b80 x1876699493969664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761675.821097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493969664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761675.821105:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761675.821108:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761675.821112:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761675.821118:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761672/real 1789761675] req@ffff8eb650f81f80 x1876699493969792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761675.821137:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493969792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761675.821145:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761675.821149:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761675.821152:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493969792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761675.821160:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761672/real 1789761675] req@ffff8eb650f83800 x1876699493969536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761675.821173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493969536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761675.821182:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761675.821186:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761675.821189:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493969536:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761676.524593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761677.548608:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761677.868570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761677.868581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761677.868586:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761677.868597:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1682 00000100:00080000:1.0:1789761677.868608:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761677.868617:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761677.868664:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493970048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761677.868694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761677.868699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761677.868701:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761677.868706:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1682 00000100:00080000:1.0:1789761677.868711:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761677.868716:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761677.868728:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761677.868741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493970048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761677.868748:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493970176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761677.868762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761677.868766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761677.868768:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761677.868773:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1682 00000100:00080000:1.0:1789761677.868783:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761677.868789:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761677.868795:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761677.868810:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493970304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761677.868822:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761677.868826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761677.868828:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761677.868835:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1682 00000800:00000100:2.0:1789761677.868836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761677.868843:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761677.868850:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761677.868857:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761677.868870:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493970432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761677.868901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761677/real 1789761677] req@ffff8eb65016ed80 x1876699493970048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761677.868926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493970048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761677.868939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761677.868943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761677.868947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493970048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761677.868981:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761677.869025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761677.869042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761677/real 1789761677] req@ffff8eb65016f480 x1876699493970176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761677.869066:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493970176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761677.869077:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761677.869080:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761677.869090:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789761677.869098:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789761677.869120:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761677.869129:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761677/real 1789761677] req@ffff8eb65016dc00 x1876699493970304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761677.869141:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493970304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761677.869151:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761677.869155:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761677.869167:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789761677.869173:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493970432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789761677.869196:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761677.869208:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761677/real 1789761677] req@ffff8eb65016df80 x1876699493970432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761677.869223:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493970432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761677.869232:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761677.869235:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789761677.869239:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493970432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761678.572611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761679.596584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761680.620618:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761681.644540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761682.668547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761682.988496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761682.988506:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761682.988511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761682.988522:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1687 00000100:00080000:2.0:1789761682.988533:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761682.988540:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761682.988590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493970560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761682.988616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761682.988619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761682.988621:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761682.988624:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1687 00000100:00080000:2.0:1789761682.988627:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761682.988629:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761682.988639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761682.988649:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493970688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761682.988662:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493970560:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761682.988665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761682.988670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761682.988672:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761682.988676:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1687 00000100:00080000:2.0:1789761682.988678:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761682.988680:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761682.988683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761682.988692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493970816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761682.988699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761682.988702:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761682.988704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761682.988707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1687 00000100:00080000:2.0:1789761682.988710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761682.988713:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761682.988716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761682.988726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493970944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761682.988737:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761682.988786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761682/real 1789761682] req@ffff8eb650f83800 x1876699493970560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761682.988804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493970560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761682.988813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761682.988816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761682.988820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493970560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761682.988841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761682.988857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761682.988866:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761682/real 1789761682] req@ffff8eb650f81f80 x1876699493970688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761682.988875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493970688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761682.988881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761682.988884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761682.988888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761682.988895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970816:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761682.988918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761682.988931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761682/real 1789761682] req@ffff8eb650f83b80 x1876699493970816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761682.988945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493970816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761682.988955:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761682.988959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761682.988963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493970816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761682.988970:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493970944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761682.988992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761682.989004:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761682/real 1789761682] req@ffff8eb650f81180 x1876699493970944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761682.989016:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493970944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761682.989023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761682.989025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761682.989028:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493970944:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761683.692522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761684.716536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761685.740561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761686.764553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761687.788563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761688.108524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761688.108537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761688.108543:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761688.108555:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1692 00000100:00080000:2.0:1789761688.108568:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761688.108576:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761688.108624:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493971072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761688.108654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761688.108659:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761688.108661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761688.108667:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1692 00000100:00080000:2.0:1789761688.108672:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761688.108677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761688.108690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761688.108691:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493971072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761688.108712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493971200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761688.108726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761688.108730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761688.108733:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761688.108737:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1692 00000100:00080000:2.0:1789761688.108743:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761688.108747:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761688.108753:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761688.108767:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493971328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761688.108776:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761688.108785:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761688.108790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761688.108792:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761688.108798:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1692 00000100:00080000:2.0:1789761688.108810:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761688.108815:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761688.108822:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761688.108832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761688/real 1789761688] req@ffff8eb650f82a00 x1876699493971072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761688.108837:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493971456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761688.108870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493971072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761688.108881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761688.108885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761688.108889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493971072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761688.108916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761688.108943:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761688.108957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761688/real 1789761688] req@ffff8eb650f83480 x1876699493971200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761688.108967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493971200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761688.108978:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761688.108980:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761688.108983:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761688.108987:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761688.109004:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761688.109011:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761688/real 1789761688] req@ffff8eb650f80e00 x1876699493971328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761688.109019:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493971328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761688.109025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761688.109027:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761688.109029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761688.109032:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493971456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761688.109042:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761688.109048:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761688/real 1789761688] req@ffff8eb650f80700 x1876699493971456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761688.109055:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493971456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761688.109061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761688.109062:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761688.109065:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493971456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761688.812585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761689.836522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761690.860543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761691.884613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761692.908531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761693.228544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761693.228557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761693.228562:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761693.228573:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1697 00000100:00080000:2.0:1789761693.228585:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761693.228593:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761693.228637:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493971584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761693.228667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761693.228672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761693.228674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761693.228680:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1697 00000100:00080000:2.0:1789761693.228684:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761693.228689:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761693.228700:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761693.228704:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493971584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761693.228719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493971712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761693.228734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761693.228738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761693.228741:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761693.228744:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1697 00000100:00080000:2.0:1789761693.228749:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761693.228754:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761693.228761:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761693.228773:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493971840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761693.228782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761693.228785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761693.228786:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761693.228789:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1697 00000800:00000100:0.0:1789761693.228795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761693.228795:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761693.228803:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761693.228807:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761693.228816:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493971968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761693.228860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761693/real 1789761693] req@ffff8eb650f81c00 x1876699493971584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761693.228884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493971584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761693.228898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761693.228903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761693.228907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493971584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761693.228935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761693.228958:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761693.228969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761693/real 1789761693] req@ffff8eb650f82680 x1876699493971712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761693.228983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493971712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761693.228991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761693.228995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761693.228998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761693.229003:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761693.229018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761693.229026:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761693/real 1789761693] req@ffff8eb650f82d80 x1876699493971840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761693.229034:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493971840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761693.229039:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761693.229042:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761693.229044:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493971840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761693.229047:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493971968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761693.229058:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761693.229067:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761693/real 1789761693] req@ffff8eb650f81880 x1876699493971968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761693.229080:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493971968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761693.229088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761693.229092:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761693.229095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493971968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761693.932531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761694.956560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761695.980577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761697.004536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761698.028541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761698.348513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761698.348525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761698.348530:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761698.348540:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1702 00000100:00080000:2.0:1789761698.348554:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761698.348563:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761698.348613:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493972096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761698.348651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761698.348653:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761698.348655:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761698.348659:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1702 00000100:00080000:2.0:1789761698.348662:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761698.348666:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761698.348676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761698.348693:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493972224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761698.348701:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493972096:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761698.348705:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761698.348711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761698.348713:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761698.348716:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1702 00000100:00080000:2.0:1789761698.348718:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761698.348721:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761698.348724:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761698.348734:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493972352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761698.348742:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761698.348746:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761698.348747:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761698.348750:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1702 00000100:00080000:2.0:1789761698.348753:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761698.348756:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761698.348759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761698.348766:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493972480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761698.348790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761698.348832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761698/real 1789761698] req@ffff8eb650f81880 x1876699493972096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761698.348852:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493972096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761698.348861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761698.348866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761698.348870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493972096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761698.348899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761698.348926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761698.348941:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761698/real 1789761698] req@ffff8eb650f82d80 x1876699493972224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761698.348955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493972224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761698.348965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761698.348969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761698.348972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761698.348977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972352:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761698.348995:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761698.349003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761698/real 1789761698] req@ffff8eb650f82680 x1876699493972352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761698.349012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493972352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761698.349022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761698.349024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761698.349026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761698.349030:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493972480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761698.349041:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761698.349047:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761698/real 1789761698] req@ffff8eb650f81c00 x1876699493972480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761698.349054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493972480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761698.349060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761698.349062:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761698.349064:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493972480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761699.052605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761700.076618:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761701.100647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761702.124588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761703.148570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761703.468503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761703.468515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761703.468519:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761703.468531:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1707 00000100:00080000:1.0:1789761703.468545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761703.468555:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761703.468605:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493972608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761703.468634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761703.468638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761703.468640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761703.468645:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1707 00000100:00080000:1.0:1789761703.468656:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761703.468662:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761703.468675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761703.468693:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493972608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761703.468694:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493972736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761703.468712:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761703.468717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761703.468719:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761703.468724:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1707 00000100:00080000:1.0:1789761703.468729:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761703.468733:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761703.468740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761703.468751:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493972864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761703.468759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761703.468762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761703.468763:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761703.468766:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1707 00000100:00080000:1.0:1789761703.468769:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761703.468772:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761703.468775:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761703.468777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761703.468787:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493972992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761703.468833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761703/real 1789761703] req@ffff8eb65016c700 x1876699493972608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761703.468853:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493972608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761703.468862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761703.468866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761703.468869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493972608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761703.468892:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761703.468911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761703.468920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761703/real 1789761703] req@ffff8eb65016ce00 x1876699493972736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761703.468929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493972736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761703.468936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761703.468938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761703.468941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761703.468945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972864:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761703.468959:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761703.468965:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761703/real 1789761703] req@ffff8eb65016c380 x1876699493972864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761703.468973:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493972864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761703.468978:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761703.468980:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761703.468982:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493972864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761703.468985:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493972992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761703.468996:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761703.469002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761703/real 1789761703] req@ffff8eb65016fb80 x1876699493972992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761703.469014:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493972992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761703.469022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761703.469026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761703.469029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493972992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761704.172614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761705.196540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761706.220589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761707.244535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761708.268544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761708.588506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761708.588522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761708.588528:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761708.588542:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1713 00000100:00080000:1.0:1789761708.588557:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761708.588566:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761708.588607:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493973120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761708.588635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761708.588638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761708.588640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761708.588645:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1713 00000100:00080000:1.0:1789761708.588649:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761708.588655:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761708.588667:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761708.588684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493973248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761708.588689:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493973120:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761708.588705:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761708.588710:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761708.588712:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761708.588718:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1713 00000100:00080000:1.0:1789761708.588723:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761708.588728:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761708.588733:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761708.588745:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493973376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761708.588754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761708.588756:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761708.588758:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761708.588762:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1713 00000100:00080000:1.0:1789761708.588766:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761708.588769:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761708.588773:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761708.588779:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761708.588783:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493973504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761708.588827:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761708/real 1789761708] req@ffff8eb65016d500 x1876699493973120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761708.588843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493973120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761708.588853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761708.588856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761708.588859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493973120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761708.588883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761708.588900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761708.588909:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761708/real 1789761708] req@ffff8eb65016e300 x1876699493973248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761708.588922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493973248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761708.588929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761708.588931:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761708.588933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761708.588937:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761708.588949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761708.588955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761708/real 1789761708] req@ffff8eb65016f800 x1876699493973376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761708.588963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493973376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761708.588968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761708.588970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761708.588972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761708.588976:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493973504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761708.588987:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761708.588993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761708/real 1789761708] req@ffff8eb65016e680 x1876699493973504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761708.589000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493973504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761708.589010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761708.589012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761708.589014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493973504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761709.292530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761710.316538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761711.340621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761712.364516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761713.388617:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761713.708511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761713.708522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761713.708527:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761713.708538:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1718 00000100:00080000:2.0:1789761713.708550:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761713.708557:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761713.708601:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493973632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761713.708626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761713.708629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761713.708630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761713.708634:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1718 00000100:00080000:2.0:1789761713.708636:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761713.708638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761713.708647:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761713.708657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493973760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761713.708665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761713.708668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761713.708670:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761713.708672:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1718 00000100:00080000:2.0:1789761713.708674:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761713.708678:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761713.708682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493973632:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761713.708683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761713.708694:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493973888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761713.708702:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761713.708705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761713.708706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761713.708709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1718 00000100:00080000:2.0:1789761713.708712:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761713.708715:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761713.708718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761713.708726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493974016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761713.708756:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761713.708796:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761713/real 1789761713] req@ffff8eb650f80700 x1876699493973632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761713.708811:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493973632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761713.708820:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761713.708823:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761713.708827:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493973632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761713.708832:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761713.708847:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761713.708855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761713/real 1789761713] req@ffff8eb650f80e00 x1876699493973760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761713.708862:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493973760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761713.708868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761713.708871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761713.708873:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761713.708895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973888:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761713.708907:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761713.708914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761713/real 1789761713] req@ffff8eb650f83480 x1876699493973888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761713.708922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493973888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761713.708927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761713.708929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761713.708935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493973888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761713.708939:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493974016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761713.708951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761713.708958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761713/real 1789761713] req@ffff8eb650f82a00 x1876699493974016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761713.708965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493974016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761713.708970:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761713.708972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761713.708974:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493974016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761714.412588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761715.436591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761716.460536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761717.484603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761718.508575:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761718.828472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761718.828483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761718.828488:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761718.828500:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1723 00000100:00080000:2.0:1789761718.828512:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761718.828520:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761718.828568:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493974144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761718.828599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761718.828602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761718.828604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761718.828607:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1723 00000100:00080000:2.0:1789761718.828610:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761718.828612:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761718.828622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761718.828632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493974272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761718.828640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761718.828643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761718.828644:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761718.828647:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1723 00000100:00080000:2.0:1789761718.828650:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761718.828652:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761718.828657:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761718.828664:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493974400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761718.828672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761718.828675:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761718.828676:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761718.828679:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1723 00000100:00080000:2.0:1789761718.828682:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761718.828687:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761718.828690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761718.828697:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493974144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761718.828699:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493974528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761718.828782:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761718.828829:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761718/real 1789761718] req@ffff8eb650f81180 x1876699493974144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761718.828850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493974144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761718.828861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761718.828865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761718.828868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493974144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761718.828874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761718.828893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761718.828902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761718/real 1789761718] req@ffff8eb650f83b80 x1876699493974272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761718.828911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493974272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761718.828917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761718.828920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761718.828922:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761718.828926:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761718.828938:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761718.828944:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761718/real 1789761718] req@ffff8eb650f81f80 x1876699493974400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761718.828952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493974400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761718.828958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761718.828960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761718.828961:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761718.828982:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493974528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761718.828993:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761718.829000:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761718/real 1789761718] req@ffff8eb650f83800 x1876699493974528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761718.829007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493974528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761718.829012:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761718.829014:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761718.829016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493974528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761719.532624:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761720.556532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761721.580621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761722.604532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761723.628537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761723.948463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761723.948474:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761723.948478:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761723.948490:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1728 00000100:00080000:2.0:1789761723.948502:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761723.948509:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761723.948551:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493974656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761723.948579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761723.948582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761723.948584:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761723.948587:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1728 00000100:00080000:2.0:1789761723.948589:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761723.948592:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761723.948601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761723.948611:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493974784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761723.948621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761723.948624:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761723.948625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761723.948630:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1728 00000100:00100000:3.0:1789761723.948631:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493974656:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761723.948633:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761723.948639:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761723.948644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761723.948652:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493974912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761723.948659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761723.948662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761723.948663:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761723.948666:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1728 00000100:00080000:2.0:1789761723.948668:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761723.948670:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761723.948673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761723.948683:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493975040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761723.948705:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761723.948746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761723/real 1789761723] req@ffff8eb650f80a80 x1876699493974656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761723.948761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493974656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761723.948771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761723.948774:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761723.948778:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493974656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761723.948798:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761723.948814:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761723.948822:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761723/real 1789761723] req@ffff8eb650f80000 x1876699493974784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761723.948830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493974784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761723.948835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761723.948837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761723.948839:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761723.948844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974912:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761723.948855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761723.948862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761723/real 1789761723] req@ffff8eb650f80380 x1876699493974912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761723.948869:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493974912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761723.948875:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761723.948877:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761723.948878:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493974912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761723.948883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493975040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761723.948894:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761723.948900:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761723/real 1789761723] req@ffff8eb650f81500 x1876699493975040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761723.948907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493975040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761723.948912:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761723.948914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761723.948916:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493975040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761724.652555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761725.676559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761726.700531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761727.724543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761728.748512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761729.068521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761729.068536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761729.068541:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761729.068552:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1733 00000100:00080000:1.0:1789761729.068565:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761729.068573:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761729.068617:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493975168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761729.068648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761729.068651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761729.068653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761729.068656:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1733 00000100:00080000:1.0:1789761729.068659:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761729.068662:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761729.068671:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761729.068683:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493975296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761729.068691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761729.068695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761729.068696:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761729.068700:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1733 00000100:00080000:1.0:1789761729.068702:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761729.068705:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761729.068708:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761729.068716:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493975168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761729.068718:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493975424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761729.068731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761729.068733:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761729.068734:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761729.068737:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1733 00000100:00080000:1.0:1789761729.068739:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761729.068741:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761729.068744:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761729.068752:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493975552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761729.068807:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761729.068850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761729/real 1789761729] req@ffff8eb65016ea00 x1876699493975168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761729.068864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493975168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761729.068873:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761729.068876:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761729.068879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493975168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761729.068885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761729.068908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761729.068922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761729/real 1789761729] req@ffff8eb65016d880 x1876699493975296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761729.068935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493975296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761729.068957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761729.068962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761729.068966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761729.068993:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761729.069018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761729.069031:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761729/real 1789761729] req@ffff8eb65016e680 x1876699493975424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761729.069044:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493975424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761729.069053:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761729.069057:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761729.069060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761729.069067:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493975552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761729.069088:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761729.069097:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761729/real 1789761729] req@ffff8eb65016f800 x1876699493975552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761729.069116:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493975552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761729.069122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761729.069124:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761729.069126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493975552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761729.772552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761730.796603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761731.820554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761732.844544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761733.868518:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761734.188544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761734.188559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761734.188565:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761734.188580:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1738 00000100:00080000:2.0:1789761734.188596:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761734.188607:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761734.188660:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493975680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761734.188696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761734.188700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761734.188703:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761734.188709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1738 00000100:00080000:2.0:1789761734.188713:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761734.188718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761734.188731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761734.188747:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493975680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761734.188750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493975808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761734.188766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761734.188770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761734.188772:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761734.188776:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1738 00000100:00080000:2.0:1789761734.188781:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761734.188785:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761734.188790:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761734.188802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493975936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761734.188813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761734.188817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789761734.188821:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761734.188822:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761734.188828:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1738 00000100:00080000:2.0:1789761734.188832:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761734.188836:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761734.188840:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761734.188849:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493976064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761734.188871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761734/real 1789761734] req@ffff8eb650f81500 x1876699493975680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761734.188888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493975680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761734.188897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761734.188900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761734.188903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493975680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761734.188943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761734.188963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761734.188972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761734/real 1789761734] req@ffff8eb650f80380 x1876699493975808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761734.188981:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493975808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761734.188988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761734.188990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761734.188993:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761734.188997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761734.189009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761734.189015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761734/real 1789761734] req@ffff8eb650f80000 x1876699493975936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761734.189022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493975936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761734.189028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761734.189030:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761734.189031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493975936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761734.189035:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493976064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761734.189046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761734.189052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761734/real 1789761734] req@ffff8eb650f80a80 x1876699493976064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761734.189060:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493976064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761734.189065:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761734.189066:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761734.189069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493976064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761734.892575:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761735.916555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761736.940540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761737.964540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761738.988597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761739.308475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761739.308486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761739.308491:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761739.308503:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1743 00000100:00080000:2.0:1789761739.308516:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761739.308524:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761739.308562:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493976192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761739.308589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761739.308592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761739.308594:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761739.308597:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1743 00000100:00080000:2.0:1789761739.308599:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761739.308602:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761739.308612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761739.308626:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493976320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761739.308641:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493976192:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761739.308642:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761739.308651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761739.308654:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761739.308660:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1743 00000100:00080000:2.0:1789761739.308665:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761739.308669:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761739.308673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761739.308686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493976448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761739.308696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761739.308700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761739.308703:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761739.308707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1743 00000100:00080000:2.0:1789761739.308712:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761739.308718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761739.308724:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761739.308738:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493976576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761739.308766:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761739.308806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761739.308827:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493976576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761740.012540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761741.036512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761742.060522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761742.380635:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789761742.380672:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761742.380686:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761742.380727:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761742.380737:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761742.380746:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789761742.380776:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761739/real 1789761742] req@ffff8eb650f81f80 x1876699493976320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761742.380801:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493976320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761742.380815:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761742.380820:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761742.380824:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761742.380832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761739/real 1789761742] req@ffff8eb650f83b80 x1876699493976448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761742.380860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493976448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761742.380869:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761742.380873:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761742.380877:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761742.380884:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761739/real 1789761742] req@ffff8eb650f81180 x1876699493976576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761742.380897:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493976576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761742.380905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761742.380909:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761742.380913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493976576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761742.380920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761739/real 1789761742] req@ffff8eb650f83800 x1876699493976192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761742.380933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493976192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761742.380941:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761742.380945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761742.380948:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493976192:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789761743.084544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761744.108538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761744.428572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761744.428583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761744.428587:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761744.428598:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1748 00000100:00080000:1.0:1789761744.428609:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761744.428617:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761744.428651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493976704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761744.428687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761744.428692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761744.428694:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761744.428711:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1748 00000100:00080000:1.0:1789761744.428717:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761744.428722:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761744.428732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761744.428749:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493976832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761744.428755:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493976704:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761744.428768:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761744.428774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761744.428776:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761744.428781:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1748 00000100:00080000:1.0:1789761744.428785:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761744.428789:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761744.428796:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761744.428811:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493976960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761744.428826:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761744.428882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761744.428942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761744/real 1789761744] req@ffff8eb65016e300 x1876699493976704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761744.428965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493976704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761744.428986:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761744.428992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761744.428999:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:1.0:1789761744.429002:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761744.429011:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493976704:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761744.429018:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1748 00000100:00080000:1.0:1789761744.429031:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761744.429038:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761744.429046:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761744.429064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789761744.429072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493977088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761744.429121:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761744.429142:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761744/real 1789761744] req@ffff8eb65016d500 x1876699493976832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761744.429157:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493976832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761744.429166:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761744.429170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761744.429174:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761744.429182:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761744.429205:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761744.429217:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761744/real 1789761744] req@ffff8eb65016fb80 x1876699493976960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761744.429229:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493976960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761744.429236:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761744.429240:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761744.429243:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493976960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761744.429362:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493977088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761744.429390:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761744.429435:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761744/real 1789761744] req@ffff8eb65016c380 x1876699493977088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761744.429448:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493977088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761744.429457:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761744.429461:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761744.429464:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493977088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761745.132573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761746.156529:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761747.180525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761748.204536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761749.228554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761749.548474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761749.548482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761749.548486:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761749.548494:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1754 00000100:00080000:1.0:1789761749.548503:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761749.548509:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761749.548547:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493977216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761749.548569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761749.548573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761749.548575:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761749.548580:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1754 00000100:00080000:1.0:1789761749.548590:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761749.548595:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761749.548605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761749.548618:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493977344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761749.548618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493977216:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761749.548630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761749.548633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761749.548635:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761749.548638:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1754 00000100:00080000:1.0:1789761749.548641:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761749.548644:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761749.548649:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761749.548659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493977472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761749.548669:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761749.548671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761749.548673:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761749.548676:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761749.548677:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1754 00000100:00080000:1.0:1789761749.548680:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761749.548684:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761749.548688:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761749.548696:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493977600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761749.548715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761749/real 1789761749] req@ffff8eb65016ce00 x1876699493977216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761749.548732:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493977216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761749.548742:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761749.548746:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761749.548750:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493977216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761749.548770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761749.548808:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761749.548819:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761749/real 1789761749] req@ffff8eb65016c700 x1876699493977344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761749.548832:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493977344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761749.548838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761749.548840:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761749.548842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761749.548846:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977472:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761749.548860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761749.548866:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761749/real 1789761749] req@ffff8eb65016df80 x1876699493977472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761749.548874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493977472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761749.548879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761749.548881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761749.548883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761749.548887:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493977600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761749.548897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761749.548903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761749/real 1789761749] req@ffff8eb65016dc00 x1876699493977600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761749.548914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493977600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761749.548919:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761749.548921:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761749.548923:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493977600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761750.252556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761751.276525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761752.300651:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761753.324543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761754.348603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761754.668512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761754.668520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761754.668529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761754.668539:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1759 00000100:00080000:0.0:1789761754.668550:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761754.668555:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761754.668586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699493977728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761754.668608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761754.668611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761754.668612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761754.668615:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1759 00000100:00080000:0.0:1789761754.668617:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761754.668620:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761754.668627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761754.668635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699493977856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761754.668642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761754.668644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761754.668645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761754.668647:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1759 00000100:00080000:0.0:1789761754.668650:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761754.668652:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761754.668656:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761754.668661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699493977984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761754.668668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761754.668670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761754.668673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761754.668676:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1759 00000100:00080000:0.0:1789761754.668677:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761754.668681:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761754.668687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761754.668697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699493978112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761754.668731:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493977728:192.168.203.160@tcp:250:kworker.0 00000800:00000100:3.0:1789761754.668834:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761754.668893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761754/real 1789761754] req@ffff8eb64304ce00 x1876699493977728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761754.668915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699493977728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761754.668926:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761754.668930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761754.668933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493977728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761754.668940:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761754.668961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761754.668971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761754/real 1789761754] req@ffff8eb64304dc00 x1876699493977856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761754.668984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699493977856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761754.669000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761754.669010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761754.669015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761754.669022:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761754.669041:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761754.669050:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761754/real 1789761754] req@ffff8eb64304c380 x1876699493977984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761754.669059:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699493977984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761754.669068:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761754.669072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761754.669075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493977984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761754.669082:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493978112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761754.669104:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761754.669117:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761754/real 1789761754] req@ffff8eb64304f800 x1876699493978112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761754.669131:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699493978112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761754.669139:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761754.669142:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761754.669146:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493978112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761755.372556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761756.396636:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761757.420595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761758.444549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761759.468549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761759.788555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761759.788566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761759.788571:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761759.788580:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1764 00000100:00080000:2.0:1789761759.788590:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761759.788596:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761759.788629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493978240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761759.788655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761759.788657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761759.788659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761759.788661:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1764 00000100:00080000:2.0:1789761759.788664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761759.788666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761759.788674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761759.788682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493978368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761759.788689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761759.788694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761759.788695:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493978240:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761759.788696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761759.788703:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1764 00000100:00080000:2.0:1789761759.788705:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761759.788707:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761759.788711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761759.788733:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493978496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761759.788740:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761759.788744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761759.788745:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761759.788748:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1764 00000100:00080000:2.0:1789761759.788759:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761759.788762:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761759.788766:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761759.788771:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761759.788778:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493978624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761759.788805:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761759/real 1789761759] req@ffff8eb650f82a00 x1876699493978240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761759.788820:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493978240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761759.788827:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761759.788830:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761759.788833:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493978240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761759.788851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761759.788866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761759.788873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761759/real 1789761759] req@ffff8eb650f83480 x1876699493978368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761759.788881:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493978368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761759.788887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761759.788889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761759.788891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761759.788894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761759.788906:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761759.788912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761759/real 1789761759] req@ffff8eb650f80e00 x1876699493978496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761759.788919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493978496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761759.788924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761759.788926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761759.788928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761759.788931:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493978624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761759.788941:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761759.788946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761759/real 1789761759] req@ffff8eb650f80700 x1876699493978624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761759.788954:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493978624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761759.788959:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761759.788960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761759.788962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493978624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761760.492542:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761761.516587:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761762.540569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761763.564558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761764.588563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761764.908511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761764.908521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761764.908525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761764.908536:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1769 00000100:00080000:2.0:1789761764.908547:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761764.908554:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761764.908592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493978752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761764.908621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761764.908626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761764.908627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761764.908632:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1769 00000100:00080000:2.0:1789761764.908636:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761764.908642:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761764.908652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761764.908660:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493978752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761764.908668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493978880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761764.908695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761764.908706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761764.908709:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761764.908717:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1769 00000100:00080000:2.0:1789761764.908722:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761764.908726:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761764.908732:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761764.908740:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761764.908756:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493979008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761764.908765:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761764/real 1789761764] req@ffff8eb650f81c00 x1876699493978752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761764.908771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761764.908778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761764.908781:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761764.908784:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493978752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761764.908790:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1769 00000100:00080000:3.0:1789761764.908798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761764.908799:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761764.908805:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:2.0:1789761764.908809:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761764.908811:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493978752:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761764.908820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761764.908833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761764.908850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761764.908859:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761764/real 1789761764] req@ffff8eb650f82680 x1876699493978880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761764.908868:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493979136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761764.908873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493978880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761764.908898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761764.908903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761764.908907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493978880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761764.908912:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979008:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761764.908925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761764.908934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761764/real 1789761764] req@ffff8eb650f82d80 x1876699493979008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761764.908942:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493979008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761764.908952:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761764.908954:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761764.908957:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761764.908968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493979136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761764.908978:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761764.908984:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761764/real 1789761764] req@ffff8eb650f81880 x1876699493979136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761764.908991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493979136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761764.908996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761764.908998:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761764.909000:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493979136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761765.612534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761766.636546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761767.660539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761768.684531:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761769.708615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761770.028513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761770.028528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761770.028535:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761770.028549:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1774 00000100:00080000:1.0:1789761770.028563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761770.028572:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761770.028620:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493979264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761770.028668:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761770.028673:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761770.028675:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761770.028680:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1774 00000100:00080000:1.0:1789761770.028684:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761770.028688:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761770.028701:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761770.028715:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493979392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761770.028727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761770.028731:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761770.028733:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761770.028737:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1774 00000100:00080000:1.0:1789761770.028741:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761770.028746:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761770.028752:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761770.028769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493979520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761770.028770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493979264:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761770.028809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761770.028814:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761770.028816:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761770.028821:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1774 00000100:00080000:1.0:1789761770.028827:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761770.028832:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761770.028838:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761770.028857:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493979648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761770.028870:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761770.028924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761770/real 1789761770] req@ffff8eb65016f480 x1876699493979264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761770.028945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493979264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761770.028957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761770.028962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761770.028967:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493979264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761770.028976:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761770.029007:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761770.029022:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761770/real 1789761770] req@ffff8eb65016ed80 x1876699493979392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761770.029035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493979392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761770.029044:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761770.029047:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761770.029052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761770.029078:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761770.029100:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761770.029113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761770/real 1789761770] req@ffff8eb65016d180 x1876699493979520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761770.029136:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493979520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761770.029145:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761770.029148:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761770.029152:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761770.029159:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493979648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761770.029181:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761770.029193:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761770/real 1789761770] req@ffff8eb65016c000 x1876699493979648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761770.029206:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493979648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761770.029214:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761770.029217:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761770.029220:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493979648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761770.732554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761771.756573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761772.780533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761773.804553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761774.828516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761775.148562:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761775.148573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761775.148578:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761775.148590:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1779 00000100:00080000:1.0:1789761775.148615:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761775.148624:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761775.148679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493979776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761775.148716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761775.148719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761775.148721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761775.148725:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1779 00000100:00080000:1.0:1789761775.148729:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761775.148732:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761775.148743:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761775.148756:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493979904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761775.148766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761775.148770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761775.148770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493979776:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761775.148773:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761775.148781:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1779 00000100:00080000:1.0:1789761775.148783:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761775.148786:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761775.148789:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761775.148800:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493980032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761775.148809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761775.148812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761775.148813:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761775.148817:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1779 00000100:00080000:1.0:1789761775.148820:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761775.148823:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761775.148828:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761775.148842:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493980160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761775.148846:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761775.148891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761775/real 1789761775] req@ffff8eb65016ca80 x1876699493979776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761775.148914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493979776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761775.148923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761775.148927:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761775.148930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493979776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761775.148953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761775.148969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761775.148978:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761775/real 1789761775] req@ffff8eb65016f100 x1876699493979904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761775.148992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493979904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761775.148997:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761775.148999:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761775.149001:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493979904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761775.149006:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761775.149033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761775.149045:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761775/real 1789761775] req@ffff8eb65016c000 x1876699493980032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761775.149064:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493980032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761775.149070:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761775.149073:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761775.149075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761775.149079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493980160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761775.149096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761775.149103:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761775/real 1789761775] req@ffff8eb65016d180 x1876699493980160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761775.149111:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493980160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761775.149125:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761775.149129:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761775.149133:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493980160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761775.852524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761776.876638:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761777.900529:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761778.924544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761779.948582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761780.268463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761780.268470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761780.268474:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761780.268481:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1784 00000100:00080000:2.0:1789761780.268490:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761780.268496:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761780.268534:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493980288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761780.268556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761780.268559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761780.268575:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761780.268579:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1784 00000100:00080000:2.0:1789761780.268582:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761780.268585:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761780.268593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761780.268603:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493980416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761780.268611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761780.268616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761780.268618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493980288:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761780.268619:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761780.268627:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1784 00000100:00080000:2.0:1789761780.268631:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761780.268635:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761780.268640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761780.268654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493980544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761780.268665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761780.268668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761780.268669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761780.268673:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1784 00000800:00000100:3.0:1789761780.268677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789761780.268679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761780.268686:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761780.268692:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761780.268705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493980672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761780.268715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761780/real 1789761780] req@ffff8eb650f81880 x1876699493980288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761780.268733:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493980288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761780.268740:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761780.268743:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761780.268745:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493980288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761780.268762:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761780.268777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761780.268784:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761780/real 1789761780] req@ffff8eb650f82d80 x1876699493980416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761780.268799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493980416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761780.268805:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761780.268807:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761780.268809:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761780.268812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980544:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761780.268824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761780.268830:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761780/real 1789761780] req@ffff8eb650f82680 x1876699493980544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761780.268836:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493980544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761780.268841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761780.268842:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761780.268844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761780.268847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493980672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761780.268861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761780.268867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761780/real 1789761780] req@ffff8eb650f81c00 x1876699493980672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761780.268873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493980672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761780.268878:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761780.268880:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761780.268881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493980672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761780.972532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761781.996657:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761783.020536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761784.044584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761785.068539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761785.388515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761785.388534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761785.388541:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761785.388551:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1789 00000100:00080000:1.0:1789761785.388564:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761785.388572:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761785.388620:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493980800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761785.388644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761785.388647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761785.388648:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761785.388651:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1789 00000100:00080000:1.0:1789761785.388654:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761785.388657:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761785.388666:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761785.388677:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493980928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761785.388684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761785.388686:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761785.388687:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761785.388690:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1789 00000100:00080000:1.0:1789761785.388691:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761785.388693:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761785.388697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761785.388699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493980800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761785.388706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493981056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761785.388717:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761785.388719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761785.388721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761785.388723:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1789 00000100:00080000:1.0:1789761785.388724:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761785.388727:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761785.388729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761785.388736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493981184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761785.388774:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761785.388819:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761785/real 1789761785] req@ffff8eb65016ed80 x1876699493980800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761785.388839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493980800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761785.388851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761785.388856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761785.388861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493980800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761785.388868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761785.388889:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761785.388901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761785/real 1789761785] req@ffff8eb65016f480 x1876699493980928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761785.388913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493980928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761785.388921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761785.388924:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761785.388928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493980928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761785.388952:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761785.388967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761785.388975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761785/real 1789761785] req@ffff8eb65016dc00 x1876699493981056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761785.388994:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493981056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761785.389001:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761785.389012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761785.389015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761785.389023:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493981184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761785.389038:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761785.389047:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761785/real 1789761785] req@ffff8eb65016df80 x1876699493981184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761785.389058:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493981184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761785.389064:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761785.389067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761785.389069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493981184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761786.092558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761787.116543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761788.140536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761789.164549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761790.188581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761790.508544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761790.508552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761790.508556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761790.508565:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1795 00000100:00080000:1.0:1789761790.508575:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761790.508580:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761790.508611:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493981312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761790.508636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761790.508639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761790.508640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761790.508645:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1795 00000100:00080000:1.0:1789761790.508649:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761790.508653:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761790.508663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761790.508678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493981440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761790.508688:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493981312:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761790.508694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761790.508699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761790.508701:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761790.508706:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1795 00000100:00080000:1.0:1789761790.508711:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761790.508715:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761790.508722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761790.508738:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493981568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761790.508747:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761790.508748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761790.508752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761790.508753:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761790.508757:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1795 00000100:00080000:1.0:1789761790.508761:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761790.508765:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761790.508769:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761790.508779:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493981696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761790.508786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761790/real 1789761790] req@ffff8eb65016c700 x1876699493981312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761790.508800:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493981312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761790.508810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761790.508813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761790.508816:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493981312:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761790.508833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761790.508848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761790.508856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761790/real 1789761790] req@ffff8eb65016ce00 x1876699493981440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761790.508864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493981440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761790.508870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761790.508872:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761790.508874:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761790.508879:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981568:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761790.508899:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761790.508910:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761790/real 1789761790] req@ffff8eb65016c380 x1876699493981568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761790.508924:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493981568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761790.508933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761790.508936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761790.508938:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761790.508943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493981696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761790.508959:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761790.508967:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761790/real 1789761790] req@ffff8eb65016fb80 x1876699493981696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761790.508975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493981696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761790.508980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761790.508982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761790.508984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493981696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761791.212588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761792.236629:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761793.260626:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761794.284538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761795.308562:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761795.628544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761795.628555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761795.628560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761795.628572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1800 00000100:00080000:0.0:1789761795.628584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761795.628592:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761795.628635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493981824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761795.628671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761795.628674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761795.628675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761795.628679:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1800 00000100:00080000:0.0:1789761795.628681:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761795.628684:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761795.628695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761795.628705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493981952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761795.628713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761795.628718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761795.628719:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493981824:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761795.628720:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761795.628728:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1800 00000100:00080000:0.0:1789761795.628731:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761795.628734:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761795.628737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761795.628747:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493982080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761795.628755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761795.628758:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761795.628759:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761795.628762:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1800 00000100:00080000:0.0:1789761795.628763:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761795.628766:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761795.628769:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761795.628776:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493982208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761795.628793:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761795.628835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761795/real 1789761795] req@ffff8eb64304c000 x1876699493981824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761795.628850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493981824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761795.628859:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761795.628862:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761795.628866:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493981824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761795.628887:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761795.628904:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761795.628913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761795/real 1789761795] req@ffff8eb64304d180 x1876699493981952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761795.628921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493981952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761795.628926:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761795.628928:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761795.628931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493981952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761795.628934:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761795.628947:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761795.628953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761795/real 1789761795] req@ffff8eb64304ed80 x1876699493982080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761795.628968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493982080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761795.628974:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761795.628977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761795.628979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761795.628983:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493982208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761795.628995:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761795.629002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761795/real 1789761795] req@ffff8eb64304e300 x1876699493982208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761795.629015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493982208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761795.629020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761795.629022:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761795.629023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493982208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761796.332550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761797.356511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761798.380568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761799.404632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761800.428609:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761800.748500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761800.748511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761800.748516:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761800.748526:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1805 00000100:00080000:2.0:1789761800.748537:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761800.748545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761800.748586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493982336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761800.748611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761800.748614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761800.748615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761800.748618:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1805 00000100:00080000:2.0:1789761800.748621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761800.748623:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761800.748632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761800.748641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493982464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761800.748650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761800.748653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761800.748655:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761800.748659:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1805 00000100:00080000:2.0:1789761800.748663:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761800.748666:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761800.748670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761800.748676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493982592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761800.748686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761800.748689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761800.748690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761800.748694:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1805 00000100:00080000:2.0:1789761800.748699:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761800.748713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761800.748717:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493982336:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761800.748718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761800.748733:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493982720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761800.748790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761800.748849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761800/real 1789761800] req@ffff8eb650f80700 x1876699493982336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761800.748863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493982336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761800.748872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761800.748875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761800.748879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493982336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761800.748886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761800.748926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761800.748940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761800/real 1789761800] req@ffff8eb650f80e00 x1876699493982464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761800.748953:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493982464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761800.748969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761800.748973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761800.748976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982464:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761800.749004:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982592:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761800.749025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761800.749034:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761800/real 1789761800] req@ffff8eb650f83480 x1876699493982592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761800.749042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493982592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761800.749047:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761800.749050:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761800.749052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982592:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761800.749056:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493982720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761800.749066:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761800.749072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761800/real 1789761800] req@ffff8eb650f82a00 x1876699493982720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761800.749078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493982720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761800.749083:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761800.749085:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761800.749088:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493982720:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761801.452637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761802.476549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761803.500569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761804.524539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761805.548530:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761805.868505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761805.868516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761805.868521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761805.868532:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1810 00000100:00080000:0.0:1789761805.868543:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761805.868550:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761805.868596:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699493982848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761805.868621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761805.868624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761805.868626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761805.868629:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1810 00000100:00080000:0.0:1789761805.868631:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761805.868634:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761805.868644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761805.868653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699493982976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761805.868659:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493982848:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789761805.868667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761805.868670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761805.868672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761805.868676:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1810 00000100:00080000:0.0:1789761805.868677:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761805.868679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761805.868683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761805.868692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699493983104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761805.868699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761805.868701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761805.868702:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761805.868705:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1810 00000100:00080000:0.0:1789761805.868708:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761805.868711:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789761805.868714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761805.868721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699493983232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761805.868786:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761805.868812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761805.868827:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493983232:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761806.572546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761807.596589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761808.620568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761808.940713:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789761808.940768:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761808.940790:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761808.940862:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761808.940874:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789761808.940885:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789761808.941309:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761805/real 1789761808] req@ffff8eb64304ed80 x1876699493982976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761808.941341:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699493982976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761808.941368:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761808.941374:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761808.941380:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493982976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761808.941561:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761805/real 1789761808] req@ffff8eb64304d180 x1876699493983104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761808.941581:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699493983104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761808.941591:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761808.941595:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761808.941600:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761808.941608:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761805/real 1789761808] req@ffff8eb64304c000 x1876699493983232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761808.941621:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699493983232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761808.941637:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761808.941642:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761808.941645:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493983232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761808.941653:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761805/real 1789761808] req@ffff8eb64304e300 x1876699493982848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761808.941667:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699493982848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761808.941676:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761808.941680:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761808.941683:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493982848:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761809.644559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761810.668604:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761810.988563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761810.988574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761810.988578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761810.988589:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1815 00000100:00080000:3.0:1789761810.988599:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761810.988620:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761810.988677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493983360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761810.988719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761810.988723:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761810.988725:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761810.988730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1815 00000100:00080000:3.0:1789761810.988736:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761810.988741:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761810.988754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761810.988771:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493983360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761810.988783:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493983488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761810.988796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761810.988801:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761810.988802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761810.988807:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1815 00000100:00080000:3.0:1789761810.988817:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761810.988822:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761810.988830:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761810.988843:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493983616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761810.988853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761810.988859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761810.988861:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761810.988865:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761810.988872:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1815 00000100:00080000:3.0:1789761810.988877:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761810.988882:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761810.988889:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761810.988900:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493983744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761810.988923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761810/real 1789761810] req@ffff8eb64514a680 x1876699493983360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761810.988947:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493983360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761810.988962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761810.988967:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761810.988972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493983360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761810.989001:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761810.989025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761810.989038:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761810/real 1789761810] req@ffff8eb645148a80 x1876699493983488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761810.989053:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493983488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761810.989063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761810.989067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761810.989070:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983488:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761810.989077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983616:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761810.989096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761810.989107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761810/real 1789761810] req@ffff8eb645149500 x1876699493983616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761810.989120:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493983616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761810.989129:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761810.989133:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761810.989136:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493983616:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761810.989143:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493983744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761810.989161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761810.989171:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761810/real 1789761810] req@ffff8eb645149c00 x1876699493983744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761810.989184:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493983744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761810.989193:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761810.989196:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761810.989200:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493983744:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761811.692524:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761812.716581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761813.740596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761814.764584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761815.788537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761816.108465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761816.108475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761816.108479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761816.108490:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1820 00000100:00080000:3.0:1789761816.108501:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761816.108508:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761816.108549:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493983872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761816.108575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761816.108578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761816.108579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761816.108587:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1820 00000100:00080000:3.0:1789761816.108590:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761816.108593:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761816.108602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761816.108612:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493984000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761816.108627:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493983872:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761816.108628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761816.108637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761816.108638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761816.108642:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1820 00000100:00080000:3.0:1789761816.108644:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761816.108647:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761816.108651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761816.108661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493984128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761816.108668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761816.108671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761816.108672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761816.108676:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1820 00000100:00080000:3.0:1789761816.108679:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761816.108681:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761816.108684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761816.108692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493984256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761816.108706:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761816.108749:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761816/real 1789761816] req@ffff8eb645148380 x1876699493983872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761816.108764:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493983872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761816.108773:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761816.108776:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761816.108780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493983872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761816.108804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761816.108826:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761816.108840:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761816/real 1789761816] req@ffff8eb645148e00 x1876699493984000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761816.108855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493984000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761816.108864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761816.108867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761816.108870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761816.108875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984128:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761816.108892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761816.108899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761816/real 1789761816] req@ffff8eb64514a300 x1876699493984128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761816.108907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493984128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761816.108913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761816.108916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761816.108918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761816.108921:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493984256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761816.108931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761816.108937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761816/real 1789761816] req@ffff8eb64514b800 x1876699493984256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761816.108945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493984256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761816.108950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761816.108952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761816.108954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493984256:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761816.812592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761817.836534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761818.860556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761819.884611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761820.908588:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761821.228536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761821.228545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761821.228549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761821.228557:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1825 00000100:00080000:3.0:1789761821.228578:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761821.228584:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761821.228616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493984384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761821.228652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761821.228654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761821.228656:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761821.228659:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1825 00000100:00080000:3.0:1789761821.228661:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761821.228663:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761821.228671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761821.228679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493984512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761821.228687:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761821.228690:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761821.228691:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761821.228693:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1825 00000100:00080000:3.0:1789761821.228695:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761821.228697:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761821.228701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761821.228704:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493984384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761821.228711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493984640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761821.228724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761821.228726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761821.228727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761821.228730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1825 00000100:00080000:3.0:1789761821.228731:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761821.228741:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761821.228744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761821.228751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493984768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761821.228773:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761821.228807:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761821/real 1789761821] req@ffff8eb645148000 x1876699493984384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761821.228840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493984384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761821.228850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761821.228854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761821.228858:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493984384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761821.228865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761821.228885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761821.228896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761821/real 1789761821] req@ffff8eb64514bb80 x1876699493984512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761821.228914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493984512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761821.228921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761821.228924:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761821.228927:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761821.228949:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984640:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761821.228964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761821.228974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761821/real 1789761821] req@ffff8eb64514b100 x1876699493984640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761821.228984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493984640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761821.228991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761821.228994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761821.228997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493984640:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761821.229012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493984768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761821.229025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761821.229034:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761821/real 1789761821] req@ffff8eb645149880 x1876699493984768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761821.229051:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493984768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761821.229071:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761821.229086:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761821.229107:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493984768:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761821.932521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761822.956589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761823.980529:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761825.004571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761826.028600:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761826.348506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761826.348517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761826.348521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761826.348532:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1830 00000100:00080000:3.0:1789761826.348563:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761826.348572:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761826.348650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493984896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761826.348699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761826.348702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761826.348703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761826.348707:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1830 00000100:00080000:3.0:1789761826.348711:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761826.348714:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761826.348724:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761826.348735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493985024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761826.348743:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761826.348746:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761826.348748:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761826.348750:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1830 00000100:00080000:3.0:1789761826.348753:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761826.348758:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761826.348760:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493984896:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789761826.348761:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761826.348771:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493985152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761826.348778:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761826.348781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761826.348782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761826.348784:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1830 00000100:00080000:3.0:1789761826.348787:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761826.348789:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761826.348792:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761826.348799:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493985280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761826.348857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761826.348908:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761826/real 1789761826] req@ffff8eb64514ad80 x1876699493984896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761826.348929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493984896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761826.348938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761826.348941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761826.348945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493984896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761826.348951:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761826.348972:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761826.348985:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761826/real 1789761826] req@ffff8eb64514b480 x1876699493985024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761826.348998:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493985024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761826.349015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761826.349019:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761826.349023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985024:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761826.349048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985152:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761826.349066:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761826.349073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761826/real 1789761826] req@ffff8eb645148700 x1876699493985152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761826.349082:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493985152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761826.349088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761826.349090:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761826.349092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985152:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761826.349095:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493985280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761826.349106:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761826.349112:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761826/real 1789761826] req@ffff8eb645149180 x1876699493985280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761826.349119:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493985280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761826.349123:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761826.349125:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761826.349128:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493985280:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761827.052545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761828.076521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761829.100519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761830.124574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761831.148546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761831.468675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761831.468690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761831.468695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761831.468709:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1835 00000100:00080000:2.0:1789761831.468723:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761831.468734:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761831.468793:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493985408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761831.468820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761831.468823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761831.468824:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761831.468828:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1835 00000100:00080000:2.0:1789761831.468831:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761831.468834:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761831.468844:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761831.468854:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493985536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761831.468862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761831.468870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761831.468871:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761831.468875:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1835 00000100:00080000:2.0:1789761831.468877:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761831.468880:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761831.468884:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761831.468892:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493985664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761831.468902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761831.468906:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493985408:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761831.468906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761831.468909:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761831.468914:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1835 00000100:00080000:2.0:1789761831.468916:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761831.468920:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761831.468929:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761831.468939:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493985792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761831.469006:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761831.469051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761831/real 1789761831] req@ffff8eb650f83800 x1876699493985408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761831.469072:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493985408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761831.469084:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761831.469088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761831.469101:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493985408:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761831.469111:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761831.469137:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761831.469152:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761831/real 1789761831] req@ffff8eb650f81180 x1876699493985536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761831.469166:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493985536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761831.469176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761831.469193:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761831.469197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985536:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761831.469225:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985664:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761831.469250:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761831.469263:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761831/real 1789761831] req@ffff8eb650f83b80 x1876699493985664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761831.469277:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493985664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761831.469286:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761831.469289:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761831.469293:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493985664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761831.469300:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493985792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761831.469322:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761831.469334:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761831/real 1789761831] req@ffff8eb650f81f80 x1876699493985792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761831.469346:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493985792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761831.469355:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761831.469359:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761831.469362:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493985792:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761832.172530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761833.196525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761834.220588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761835.244583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761836.268608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761836.588513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761836.588525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761836.588531:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761836.588545:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1841 00000100:00080000:1.0:1789761836.588560:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761836.588569:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761836.588621:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493985920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761836.588646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761836.588650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761836.588652:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761836.588657:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1841 00000100:00080000:1.0:1789761836.588662:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761836.588667:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761836.588681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761836.588689:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493985920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761836.588700:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493986048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761836.588715:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761836.588719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761836.588721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761836.588727:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1841 00000100:00080000:1.0:1789761836.588732:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761836.588737:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761836.588743:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761836.588757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493986176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761836.588770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761836.588774:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761836.588776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761836.588780:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761836.588786:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1841 00000100:00080000:1.0:1789761836.588792:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761836.588797:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761836.588803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761836.588817:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493986304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761836.588835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761836/real 1789761836] req@ffff8eb65016d500 x1876699493985920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761836.588853:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493985920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761836.588864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761836.588868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761836.588871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493985920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761836.588894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761836.588912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761836.588921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761836/real 1789761836] req@ffff8eb65016e300 x1876699493986048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761836.588929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493986048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761836.588935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761836.588938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761836.588940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986048:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761836.588944:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986176:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761836.588956:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761836.588962:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761836/real 1789761836] req@ffff8eb65016f800 x1876699493986176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761836.588975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493986176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761836.588980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761836.588982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761836.588984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986176:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761836.588988:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493986304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761836.588998:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761836.589012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761836/real 1789761836] req@ffff8eb65016e680 x1876699493986304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761836.589023:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493986304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761836.589028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761836.589031:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761836.589033:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493986304:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761837.292645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761838.316635:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761839.340561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761840.364544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761841.388523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761841.708471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761841.708479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761841.708483:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761841.708492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1846 00000100:00080000:1.0:1789761841.708501:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761841.708507:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761841.708539:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699493986432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761841.708563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761841.708567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761841.708568:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761841.708572:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1846 00000100:00080000:1.0:1789761841.708574:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761841.708576:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761841.708584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761841.708593:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699493986560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761841.708601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761841.708604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761841.708605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761841.708609:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493986432:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761841.708617:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1846 00000100:00080000:1.0:1789761841.708620:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761841.708622:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761841.708626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761841.708637:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699493986688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761841.708644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761841.708647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761841.708648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761841.708651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1846 00000100:00080000:1.0:1789761841.708654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761841.708657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761841.708660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761841.708665:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761841.708670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699493986816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761841.708706:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761841/real 1789761841] req@ffff8eb65016d880 x1876699493986432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761841.708734:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699493986432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761841.708744:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761841.708747:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761841.708750:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493986432:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761841.708770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761841.708788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761841.708797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761841/real 1789761841] req@ffff8eb65016ea00 x1876699493986560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761841.708806:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699493986560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761841.708811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761841.708814:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761841.708816:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986560:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761841.708819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986688:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761841.708830:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761841.708836:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761841/real 1789761841] req@ffff8eb65016e680 x1876699493986688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761841.708843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699493986688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761841.708853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761841.708855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761841.708858:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493986688:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761841.708861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493986816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761841.708871:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761841.708876:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761841/real 1789761841] req@ffff8eb65016f800 x1876699493986816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761841.708883:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699493986816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761841.708888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761841.708890:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761841.708892:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493986816:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761842.412529:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761843.436573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761844.460566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761845.484539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761846.508545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761846.828526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761846.828538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761846.828543:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761846.828553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1851 00000100:00080000:2.0:1789761846.828565:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761846.828573:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761846.828610:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493986944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761846.828631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761846.828633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761846.828635:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761846.828638:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1851 00000100:00080000:2.0:1789761846.828641:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761846.828643:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761846.828650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761846.828659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493987072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761846.828667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761846.828670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761846.828671:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761846.828673:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1851 00000100:00080000:2.0:1789761846.828676:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761846.828681:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761846.828682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493986944:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789761846.828685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761846.828696:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493987200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761846.828703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761846.828705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761846.828706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761846.828708:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1851 00000100:00080000:2.0:1789761846.828710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761846.828712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761846.828716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761846.828723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493987328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761846.828745:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761846.828783:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761846/real 1789761846] req@ffff8eb650f80a80 x1876699493986944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761846.828796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493986944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761846.828809:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761846.828812:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761846.828816:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493986944:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761846.828823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761846.828838:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761846.828846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761846/real 1789761846] req@ffff8eb650f80000 x1876699493987072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761846.828854:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493987072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761846.828860:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761846.828862:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761846.828864:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761846.828881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987200:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761846.828892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761846.828898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761846/real 1789761846] req@ffff8eb650f80380 x1876699493987200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761846.828906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493987200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761846.828911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761846.828913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761846.828915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761846.828919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493987328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761846.828929:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761846.828934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761846/real 1789761846] req@ffff8eb650f81500 x1876699493987328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761846.828942:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493987328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761846.828947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761846.828948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761846.828950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493987328:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761847.532653:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761848.556535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761849.580626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761850.604550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761851.628586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761851.948540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761851.948551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761851.948556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761851.948567:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1856 00000100:00080000:2.0:1789761851.948580:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761851.948588:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761851.948624:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699493987456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761851.948647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761851.948650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761851.948651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761851.948654:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1856 00000100:00080000:2.0:1789761851.948657:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761851.948661:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761851.948669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761851.948678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699493987584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761851.948689:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493987456:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761851.948692:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761851.948705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761851.948708:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761851.948712:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1856 00000100:00080000:2.0:1789761851.948714:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761851.948716:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761851.948720:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761851.948731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699493987712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761851.948738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761851.948742:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761851.948743:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761851.948746:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1856 00000100:00080000:2.0:1789761851.948749:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761851.948751:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761851.948757:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761851.948764:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699493987840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761851.948786:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761851.948833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761851/real 1789761851] req@ffff8eb650f81500 x1876699493987456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761851.948849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699493987456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761851.948858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761851.948861:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761851.948865:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493987456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761851.948891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761851.948907:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761851.948915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761851/real 1789761851] req@ffff8eb650f80380 x1876699493987584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761851.948923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699493987584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761851.948929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761851.948932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761851.948934:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987584:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761851.948938:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987712:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761851.948949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761851.948954:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761851/real 1789761851] req@ffff8eb650f80000 x1876699493987712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761851.948962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699493987712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761851.948967:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761851.948969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761851.948971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493987712:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761851.948975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493987840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761851.948986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761851.948992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761851/real 1789761851] req@ffff8eb650f80a80 x1876699493987840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761851.949004:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699493987840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761851.949009:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761851.949011:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761851.949013:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493987840:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761852.652593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761853.676675:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761854.700597:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761855.724640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761856.748602:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761857.068503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761857.068511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761857.068515:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761857.068523:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1861 00000100:00080000:2.0:1789761857.068532:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761857.068538:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761857.068570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699493987968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761857.068598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761857.068600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761857.068602:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761857.068605:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1861 00000100:00080000:2.0:1789761857.068607:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761857.068610:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761857.068618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761857.068631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699493988096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761857.068644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761857.068648:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493987968:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761857.068650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761857.068655:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761857.068661:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1861 00000100:00080000:2.0:1789761857.068665:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761857.068669:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761857.068676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761857.068687:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699493988224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761857.068695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761857.068698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761857.068700:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761857.068702:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1861 00000100:00080000:2.0:1789761857.068708:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761857.068710:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761857.068715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761857.068718:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789761857.068731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699493988352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761857.068766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761857/real 1789761857] req@ffff8eb650f81f80 x1876699493987968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761857.068787:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699493987968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761857.068798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761857.068803:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761857.068807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493987968:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761857.068831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761857.068850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761857.068862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761857/real 1789761857] req@ffff8eb650f83b80 x1876699493988096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761857.068883:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699493988096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761857.068893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761857.068897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761857.068900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988096:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761857.068907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988224:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761857.068924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761857.068936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761857/real 1789761857] req@ffff8eb650f81180 x1876699493988224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761857.068949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699493988224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761857.068958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761857.068961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761857.068965:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988224:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761857.068972:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493988352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761857.069002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761857.069012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761857/real 1789761857] req@ffff8eb650f83800 x1876699493988352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761857.069020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699493988352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761857.069033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761857.069036:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761857.069040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493988352:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761857.772577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761858.796556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761859.820582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761860.844525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761861.868527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761862.188553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761862.188562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761862.188566:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761862.188577:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1866 00000100:00080000:3.0:1789761862.188589:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761862.188597:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761862.188640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493988480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761862.188664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761862.188667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761862.188668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761862.188672:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1866 00000100:00080000:3.0:1789761862.188674:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761862.188678:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761862.188688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761862.188700:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493988480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761862.188700:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493988608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761862.188715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761862.188717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761862.188718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761862.188720:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1866 00000100:00080000:3.0:1789761862.188722:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761862.188725:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761862.188728:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761862.188736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493988736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761862.188743:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761862.188745:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761862.188747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761862.188750:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1866 00000100:00080000:3.0:1789761862.188753:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761862.188755:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761862.188759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761862.188766:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493988864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761862.188778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761862.188820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761862/real 1789761862] req@ffff8eb645149180 x1876699493988480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761862.188835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493988480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761862.188843:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761862.188847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761862.188850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493988480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761862.188872:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761862.188887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761862.188895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761862/real 1789761862] req@ffff8eb645148700 x1876699493988608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761862.188904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493988608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761862.188914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761862.188917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761862.188919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761862.188923:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988736:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761862.188933:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761862.188939:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761862/real 1789761862] req@ffff8eb64514b480 x1876699493988736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761862.188946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493988736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761862.188966:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761862.188968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761862.188971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493988736:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761862.188975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493988864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761862.188990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761862.188997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761862/real 1789761862] req@ffff8eb64514ad80 x1876699493988864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761862.189005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493988864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761862.189010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761862.189012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761862.189015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493988864:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761862.892615:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761863.916576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761864.940628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761865.964650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761866.988635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761867.308589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761867.308601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761867.308606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761867.308617:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1871 00000100:00080000:2.0:1789761867.308630:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761867.308638:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761867.308677:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699493988992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761867.308707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761867.308711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761867.308713:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761867.308718:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1871 00000100:00080000:2.0:1789761867.308722:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761867.308726:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761867.308736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761867.308752:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699493989120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761867.308765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761867.308769:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761867.308772:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761867.308776:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1871 00000100:00080000:2.0:1789761867.308781:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761867.308786:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761867.308793:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761867.308799:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493988992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761867.308813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699493989248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761867.308830:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761867.308835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761867.308838:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761867.308843:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1871 00000100:00080000:2.0:1789761867.308848:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761867.308853:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761867.308860:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761867.308877:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699493989376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761867.308900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761867.308989:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761867/real 1789761867] req@ffff8eb650f82a00 x1876699493988992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761867.309016:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699493988992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761867.309034:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761867.309042:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761867.309048:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493988992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761867.309059:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761867.309095:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761867.309114:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761867/real 1789761867] req@ffff8eb650f83480 x1876699493989120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761867.309133:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699493989120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761867.309147:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761867.309151:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761867.309155:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989120:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761867.309191:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989248:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761867.309215:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761867.309226:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761867/real 1789761867] req@ffff8eb650f80e00 x1876699493989248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761867.309238:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699493989248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761867.309246:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761867.309249:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761867.309252:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989248:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761867.309258:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493989376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761867.309273:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761867.309282:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761867/real 1789761867] req@ffff8eb650f80700 x1876699493989376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761867.309292:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699493989376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761867.309300:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761867.309302:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761867.309305:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493989376:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761868.012523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761869.036576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761870.060559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761871.084517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761872.108555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761872.428530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761872.428543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761872.428549:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761872.428562:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1876 00000100:00080000:2.0:1789761872.428590:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761872.428598:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761872.428635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493989504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761872.428679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761872.428682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761872.428684:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761872.428687:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1876 00000100:00080000:2.0:1789761872.428691:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761872.428694:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761872.428703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761872.428718:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493989632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761872.428730:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761872.428734:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493989504:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789761872.428735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761872.428740:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761872.428748:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1876 00000100:00080000:2.0:1789761872.428751:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761872.428754:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761872.428759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761872.428768:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493989760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761872.428776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761872.428794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761872.428807:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761872.428810:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1876 00000100:00080000:2.0:1789761872.428821:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761872.428824:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761872.428828:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761872.428836:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493989888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761872.428896:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761872.428925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761872.428943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493989888:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761873.132571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761874.156575:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761875.180623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761875.500733:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789761875.500793:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761875.500822:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761875.500886:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761875.500912:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761875.500924:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789761875.500960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761872/real 1789761875] req@ffff8eb650f82680 x1876699493989632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761875.500995:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493989632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761875.501012:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761875.501018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761875.501023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761875.501032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761872/real 1789761875] req@ffff8eb650f82d80 x1876699493989760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761875.501045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493989760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761875.501054:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761875.501071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761875.501076:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493989760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761875.501084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761872/real 1789761875] req@ffff8eb650f81880 x1876699493989888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761875.501103:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493989888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761875.501111:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761875.501115:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761875.501119:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493989888:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761875.501126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761872/real 1789761875] req@ffff8eb650f81c00 x1876699493989504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761875.501139:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493989504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761875.501147:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761875.501151:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761875.501154:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493989504:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789761876.204554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761877.228607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761877.548503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761877.548513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761877.548517:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761877.548528:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1882 00000100:00080000:2.0:1789761877.548540:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761877.548548:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789761877.548597:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699493990016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761877.548623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761877.548626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761877.548627:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761877.548630:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1882 00000100:00080000:2.0:1789761877.548633:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761877.548635:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761877.548645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761877.548654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699493990144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761877.548661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761877.548664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761877.548665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761877.548667:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1882 00000100:00080000:2.0:1789761877.548669:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761877.548671:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761877.548675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761877.548681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699493990272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761877.548688:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761877.548691:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761877.548692:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761877.548696:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1882 00000100:00080000:2.0:1789761877.548698:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761877.548702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789761877.548706:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761877.548706:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493990016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789761877.548714:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699493990400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761877.548802:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761877.548868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761877/real 1789761877] req@ffff8eb650f81c00 x1876699493990016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761877.548890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699493990016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761877.548903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761877.548908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761877.548913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493990016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761877.548922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761877.548946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761877.548961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761877/real 1789761877] req@ffff8eb650f81880 x1876699493990144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761877.548973:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699493990144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761877.548981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761877.548984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761877.548988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990144:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761877.548994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990272:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789761877.549016:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761877.549029:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761877/real 1789761877] req@ffff8eb650f82d80 x1876699493990272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761877.549042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699493990272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761877.549050:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761877.549053:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761877.549056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990272:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761877.549091:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493990400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789761877.549112:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761877.549125:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761877/real 1789761877] req@ffff8eb650f82680 x1876699493990400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761877.549138:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699493990400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761877.549145:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761877.549148:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761877.549152:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493990400:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761878.252679:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761879.276657:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761880.300580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761881.324633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761882.348546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761882.668508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761882.668517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761882.668522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761882.668533:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1887 00000100:00080000:3.0:1789761882.668545:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761882.668553:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761882.668595:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493990528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761882.668621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761882.668623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761882.668625:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761882.668629:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1887 00000100:00080000:3.0:1789761882.668631:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761882.668635:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761882.668644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761882.668656:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493990528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761882.668657:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493990656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761882.668671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761882.668673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761882.668675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761882.668677:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1887 00000100:00080000:3.0:1789761882.668679:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761882.668681:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761882.668685:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761882.668693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493990784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761882.668701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761882.668704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761882.668705:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761882.668708:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1887 00000100:00080000:3.0:1789761882.668710:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761882.668712:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761882.668715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761882.668722:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493990912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761882.668742:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761882.668820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761882/real 1789761882] req@ffff8eb645149880 x1876699493990528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761882.668838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493990528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761882.668847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761882.668850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761882.668854:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493990528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761882.668878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761882.668897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761882.668906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761882/real 1789761882] req@ffff8eb64514b100 x1876699493990656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761882.668920:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493990656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761882.668934:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761882.668937:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761882.668939:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761882.668944:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990784:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761882.668957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761882.668964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761882/real 1789761882] req@ffff8eb64514bb80 x1876699493990784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761882.668972:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493990784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761882.668978:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761882.668980:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761882.668982:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493990784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761882.668985:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493990912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761882.668996:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761882.669002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761882/real 1789761882] req@ffff8eb645148000 x1876699493990912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761882.669009:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493990912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761882.669019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761882.669022:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761882.669024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493990912:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761883.372551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761884.396566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761885.420620:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761886.444684:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761887.468510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761887.788562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761887.788576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761887.788582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761887.788595:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1892 00000100:00080000:1.0:1789761887.788609:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761887.788618:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761887.788661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699493991040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761887.788690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761887.788695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761887.788697:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761887.788702:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1892 00000100:00080000:1.0:1789761887.788706:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761887.788711:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761887.788723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761887.788737:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699493991168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761887.788741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493991040:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761887.788748:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761887.788752:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761887.788754:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761887.788771:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1892 00000100:00080000:1.0:1789761887.788776:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761887.788779:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761887.788785:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761887.788796:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699493991296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761887.788805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761887.788809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761887.788811:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761887.788814:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1892 00000100:00080000:1.0:1789761887.788817:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761887.788821:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761887.788825:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761887.788833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761887.788834:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699493991424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761887.788890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761887/real 1789761887] req@ffff8eb65016e300 x1876699493991040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761887.788914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699493991040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761887.788924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761887.788927:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761887.788930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493991040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761887.788954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761887.788974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761887.788983:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761887/real 1789761887] req@ffff8eb65016d500 x1876699493991168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761887.788991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699493991168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761887.789001:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761887.789004:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761887.789006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761887.789010:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991296:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761887.789022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761887.789028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761887/real 1789761887] req@ffff8eb65016fb80 x1876699493991296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761887.789036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699493991296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761887.789041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761887.789043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761887.789045:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991296:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761887.789048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493991424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761887.789058:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761887.789064:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761887/real 1789761887] req@ffff8eb65016c380 x1876699493991424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761887.789072:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699493991424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761887.789077:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761887.789078:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761887.789081:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493991424:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761888.492613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761889.516526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761890.540551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761891.564568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761892.588576:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761892.908498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761892.908508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761892.908512:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761892.908524:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1897 00000100:00080000:3.0:1789761892.908535:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761892.908542:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761892.908584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493991552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761892.908609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761892.908612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761892.908614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761892.908617:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1897 00000100:00080000:3.0:1789761892.908620:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761892.908624:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761892.908634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761892.908648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493991680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761892.908660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761892.908662:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493991552:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761892.908664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761892.908669:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761892.908683:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1897 00000100:00080000:3.0:1789761892.908686:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761892.908689:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761892.908693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761892.908703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493991808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761892.908714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761892.908730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761892.908733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761892.908738:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1897 00000100:00080000:3.0:1789761892.908744:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761892.908748:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761892.908752:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761892.908753:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761892.908770:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493991936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761892.908803:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761892/real 1789761892] req@ffff8eb64514b800 x1876699493991552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761892.908819:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493991552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761892.908828:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761892.908831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761892.908834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493991552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761892.908854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761892.908871:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761892.908879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761892/real 1789761892] req@ffff8eb64514a300 x1876699493991680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761892.908888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493991680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761892.908893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761892.908896:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761892.908898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991680:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761892.908903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991808:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761892.908915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761892.908921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761892/real 1789761892] req@ffff8eb645148e00 x1876699493991808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761892.908928:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493991808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761892.908933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761892.908935:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761892.908937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493991808:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761892.908941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493991936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761892.908951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761892.908957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761892/real 1789761892] req@ffff8eb645148380 x1876699493991936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761892.908964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493991936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761892.908969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761892.908971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761892.908973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493991936:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761893.612551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761894.636585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761895.660565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761896.684554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761897.708515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761898.028508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761898.028516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761898.028521:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761898.028530:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1902 00000100:00080000:1.0:1789761898.028539:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761898.028545:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761898.028576:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699493992064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761898.028600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761898.028605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761898.028606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761898.028609:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1902 00000100:00080000:1.0:1789761898.028612:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761898.028615:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761898.028622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761898.028629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699493992192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761898.028636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761898.028639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761898.028640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761898.028642:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1902 00000100:00080000:1.0:1789761898.028644:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761898.028646:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761898.028649:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761898.028655:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493992320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761898.028662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761898.028664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761898.028665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761898.028668:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1902 00000100:00080000:1.0:1789761898.028670:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761898.028673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761898.028695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761898.028699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493992064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761898.028706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493992448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761898.028764:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761898.028808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761898/real 1789761898] req@ffff8eb65016ce00 x1876699493992064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761898.028823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699493992064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761898.028831:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761898.028834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761898.028837:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493992064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761898.028842:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761898.028860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761898.028869:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761898/real 1789761898] req@ffff8eb65016c700 x1876699493992192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761898.028876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699493992192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761898.028882:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761898.028884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761898.028886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761898.028889:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992320:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761898.028900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761898.028906:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761898/real 1789761898] req@ffff8eb65016df80 x1876699493992320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761898.028917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493992320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761898.028922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761898.028924:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761898.028926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761898.028946:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493992448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761898.028957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761898.028969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761898/real 1789761898] req@ffff8eb65016dc00 x1876699493992448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761898.028980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493992448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761898.028987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761898.028990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761898.028994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493992448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761898.732577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761899.756561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761900.780564:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761901.804553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761902.828550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761903.148512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761903.148521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761903.148524:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761903.148533:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1907 00000100:00080000:3.0:1789761903.148542:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761903.148548:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761903.148596:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493992576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761903.148624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761903.148628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761903.148631:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761903.148635:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1907 00000100:00080000:3.0:1789761903.148639:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761903.148656:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789761903.148663:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493992576:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789761903.148668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761903.148689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493992704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761903.148703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761903.148707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761903.148709:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761903.148714:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1907 00000100:00080000:3.0:1789761903.148717:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761903.148722:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761903.148724:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761903.148730:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761903.148742:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493992832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761903.148751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761903.148753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761903.148755:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761903.148759:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1907 00000100:00100000:0.0:1789761903.148762:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761903/real 1789761903] req@ffff8eb645149c00 x1876699493992576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761903.148764:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761903.148771:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761903.148784:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761903.148788:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493992576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761903.148796:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493992960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761903.148797:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761903.148806:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761903.148813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493992576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761903.148837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761903.148855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761903.148864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761903/real 1789761903] req@ffff8eb645149500 x1876699493992704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761903.148872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493992704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761903.148877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761903.148879:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761903.148881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992704:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761903.148885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992832:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761903.148902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761903.148909:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761903/real 1789761903] req@ffff8eb645148a80 x1876699493992832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761903.148918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493992832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761903.148928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761903.148930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761903.148933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493992832:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761903.148941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493992960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761903.148960:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761903.148972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761903/real 1789761903] req@ffff8eb64514a680 x1876699493992960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761903.148991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493992960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761903.148999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761903.149001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761903.149004:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493992960:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761903.852547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761904.876519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761905.900539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761906.924557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761907.948673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761908.268525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761908.268541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761908.268546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761908.268561:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1912 00000100:00080000:3.0:1789761908.268575:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761908.268585:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761908.268663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493993088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761908.268698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761908.268703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761908.268705:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761908.268710:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1912 00000100:00080000:3.0:1789761908.268715:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761908.268720:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761908.268732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761908.268751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493993216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761908.268757:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493993088:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761908.268771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761908.268776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761908.268778:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761908.268784:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1912 00000100:00080000:3.0:1789761908.268790:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761908.268795:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761908.268802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761908.268816:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493993344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761908.268829:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761908.268833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761908.268836:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761908.268837:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761908.268846:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1912 00000100:00080000:3.0:1789761908.268851:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761908.268855:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761908.268861:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761908.268874:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493993472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761908.268891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761908/real 1789761908] req@ffff8eb64514a680 x1876699493993088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761908.268914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493993088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761908.268934:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761908.268938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761908.268943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493993088:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761908.268975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761908.269001:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761908.269015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761908/real 1789761908] req@ffff8eb645148a80 x1876699493993216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761908.269025:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493993216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761908.269031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761908.269033:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761908.269035:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993216:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761908.269040:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993344:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761908.269055:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761908.269062:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761908/real 1789761908] req@ffff8eb645149500 x1876699493993344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761908.269070:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493993344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761908.269075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761908.269077:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761908.269079:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993344:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761908.269083:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493993472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761908.269094:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761908.269100:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761908/real 1789761908] req@ffff8eb645149c00 x1876699493993472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761908.269108:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493993472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761908.269113:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761908.269114:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761908.269116:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493993472:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761908.972571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761909.996606:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761911.020621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761912.044646:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761913.068600:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761913.388538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761913.388553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761913.388560:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761913.388606:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1917 00000100:00080000:1.0:1789761913.388645:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761913.388654:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761913.388698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493993600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761913.388724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761913.388727:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761913.388729:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761913.388733:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1917 00000100:00080000:1.0:1789761913.388736:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761913.388739:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761913.388750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761913.388762:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493993728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761913.388772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761913.388775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761913.388780:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761913.388782:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493993600:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761913.388786:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1917 00000100:00080000:1.0:1789761913.388793:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761913.388797:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761913.388800:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761913.388812:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493993856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761913.388820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761913.388824:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761913.388826:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761913.388829:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1917 00000100:00080000:1.0:1789761913.388849:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761913.388852:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761913.388856:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761913.388866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789761913.388872:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493993984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761913.388919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761913/real 1789761913] req@ffff8eb65016f480 x1876699493993600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761913.388937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493993600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761913.388947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761913.388950:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761913.388954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493993600:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761913.389001:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761913.389027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761913.389041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761913/real 1789761913] req@ffff8eb65016ed80 x1876699493993728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761913.389054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493993728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761913.389063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761913.389066:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761913.389070:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761913.389076:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993856:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761913.389093:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761913.389103:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761913/real 1789761913] req@ffff8eb65016d180 x1876699493993856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761913.389116:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493993856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761913.389124:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761913.389127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761913.389130:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493993856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761913.389136:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493993984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761913.389153:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761913.389163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761913/real 1789761913] req@ffff8eb65016c000 x1876699493993984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761913.389190:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493993984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761913.389199:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761913.389202:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761913.389205:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493993984:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761914.092664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761915.116677:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761916.140616:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761917.164639:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761918.188649:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761918.508521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761918.508542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761918.508547:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761918.508558:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1923 00000100:00080000:3.0:1789761918.508570:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761918.508578:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761918.508631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493994112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761918.508657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761918.508659:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761918.508661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761918.508664:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1923 00000100:00080000:3.0:1789761918.508667:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761918.508669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761918.508679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761918.508699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493994240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761918.508707:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761918.508710:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761918.508712:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761918.508715:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1923 00000100:00080000:3.0:1789761918.508716:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761918.508720:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761918.508723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761918.508729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493994368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761918.508738:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761918.508741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493994112:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761918.508742:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761918.508745:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761918.508752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1923 00000100:00080000:3.0:1789761918.508755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761918.508758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761918.508761:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761918.508768:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493994496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761918.508845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761918.508902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761918/real 1789761918] req@ffff8eb645148380 x1876699493994112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761918.508918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493994112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761918.508927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761918.508930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761918.508933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493994112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761918.508949:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761918.508966:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761918.508975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761918/real 1789761918] req@ffff8eb645148e00 x1876699493994240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761918.508988:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493994240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761918.508994:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761918.508996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761918.508998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994240:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761918.509021:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994368:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761918.509033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761918.509039:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761918/real 1789761918] req@ffff8eb64514a300 x1876699493994368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761918.509052:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493994368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761918.509060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761918.509064:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761918.509078:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994368:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761918.509085:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493994496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761918.509105:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761918.509117:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761918/real 1789761918] req@ffff8eb64514b800 x1876699493994496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761918.509125:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493994496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761918.509135:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761918.509137:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761918.509140:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493994496:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761919.212629:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761920.236653:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761921.260584:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761922.284622:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761923.308626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761923.628526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761923.628554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761923.628559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761923.628572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1928 00000100:00080000:3.0:1789761923.628584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761923.628592:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761923.628640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493994624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761923.628679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761923.628683:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761923.628684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761923.628688:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1928 00000100:00080000:3.0:1789761923.628690:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761923.628693:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761923.628704:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761923.628714:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493994752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761923.628722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761923.628726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761923.628727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761923.628731:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1928 00000100:00080000:3.0:1789761923.628733:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761923.628736:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761923.628739:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761923.628744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493994624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761923.628749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493994880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761923.628759:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761923.628762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761923.628763:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761923.628766:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1928 00000100:00080000:3.0:1789761923.628768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761923.628771:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761923.628774:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761923.628782:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493995008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761923.628865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761923.628927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761923/real 1789761923] req@ffff8eb645148000 x1876699493994624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761923.628953:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493994624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761923.628969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761923.628975:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761923.628981:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493994624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761923.628991:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761923.629033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761923.629051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761923/real 1789761923] req@ffff8eb64514bb80 x1876699493994752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761923.629067:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493994752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761923.629087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761923.629091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761923.629096:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761923.629130:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994880:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761923.629157:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761923.629185:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761923/real 1789761923] req@ffff8eb64514b100 x1876699493994880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761923.629196:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493994880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761923.629204:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761923.629208:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761923.629211:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493994880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761923.629217:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493995008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761923.629243:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761923.629255:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761923/real 1789761923] req@ffff8eb645149880 x1876699493995008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761923.629266:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493995008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761923.629290:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761923.629294:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761923.629300:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493995008:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761924.332562:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761925.356618:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761926.380606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761927.404671:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761928.428558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761928.748577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761928.748592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761928.748597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761928.748611:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1933 00000100:00080000:1.0:1789761928.748623:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761928.748632:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761928.748687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699493995136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761928.748716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761928.748722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761928.748724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761928.748727:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1933 00000100:00080000:1.0:1789761928.748730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761928.748733:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761928.748750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761928.748761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699493995264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761928.748771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761928.748774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761928.748776:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761928.748780:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1933 00000100:00080000:1.0:1789761928.748782:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761928.748785:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761928.748789:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761928.748800:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699493995392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761928.748809:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493995136:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761928.748812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761928.748819:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761928.748821:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761928.748830:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1933 00000100:00080000:1.0:1789761928.748834:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761928.748837:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761928.748842:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761928.748853:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699493995520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761928.748893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761928.748943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761928/real 1789761928] req@ffff8eb65016f100 x1876699493995136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761928.748968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699493995136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761928.748982:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761928.748988:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761928.748993:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493995136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761928.749003:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761928.749029:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761928.749052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761928/real 1789761928] req@ffff8eb65016ca80 x1876699493995264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761928.749062:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699493995264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761928.749069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761928.749072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761928.749074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761928.749103:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995392:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761928.749118:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761928.749126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761928/real 1789761928] req@ffff8eb65016c000 x1876699493995392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761928.749136:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699493995392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761928.749142:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761928.749145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761928.749147:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995392:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761928.749151:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493995520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761928.749162:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761928.749169:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761928/real 1789761928] req@ffff8eb65016d180 x1876699493995520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761928.749178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699493995520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761928.749184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761928.749186:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761928.749188:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493995520:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761929.452663:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761930.476658:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761931.500631:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761932.524645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761933.548601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761933.868509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761933.868520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761933.868525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761933.868537:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1938 00000100:00080000:3.0:1789761933.868549:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761933.868556:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761933.868598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493995648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761933.868623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761933.868626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761933.868627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761933.868631:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1938 00000100:00080000:3.0:1789761933.868633:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761933.868636:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761933.868644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761933.868654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493995776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761933.868672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761933.868681:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493995648:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761933.868682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761933.868685:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761933.868693:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1938 00000100:00080000:3.0:1789761933.868696:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761933.868698:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761933.868702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761933.868714:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493995904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761933.868737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761933.868740:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761933.868742:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761933.868746:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1938 00000100:00080000:3.0:1789761933.868748:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761933.868751:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761933.868754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761933.868761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493996032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761933.868776:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761933.868831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761933/real 1789761933] req@ffff8eb64514ad80 x1876699493995648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761933.868846:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493995648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761933.868855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761933.868859:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761933.868862:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493995648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761933.868897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761933.868912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761933.868921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761933/real 1789761933] req@ffff8eb64514b480 x1876699493995776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761933.868930:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493995776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761933.868936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761933.868938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761933.868940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995776:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761933.868944:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995904:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761933.868957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761933.868963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761933/real 1789761933] req@ffff8eb645148700 x1876699493995904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761933.868976:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493995904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761933.868981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761933.868983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761933.868985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493995904:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761933.868988:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493996032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761933.869002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761933.869013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761933/real 1789761933] req@ffff8eb645149180 x1876699493996032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761933.869025:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493996032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761933.869035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761933.869038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761933.869042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493996032:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761934.572588:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761935.596616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761936.620542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761937.644573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761938.668565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761938.988507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761938.988518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761938.988523:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761938.988534:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1943 00000100:00080000:3.0:1789761938.988545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761938.988553:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761938.988591:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493996160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761938.988615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761938.988618:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761938.988619:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761938.988622:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1943 00000100:00080000:3.0:1789761938.988625:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761938.988629:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761938.988637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761938.988646:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493996288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761938.988655:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493996160:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761938.988657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761938.988664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761938.988666:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761938.988671:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1943 00000100:00080000:3.0:1789761938.988675:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761938.988679:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761938.988685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761938.988728:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493996416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761938.988740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761938.988743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761938.988745:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761938.988750:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1943 00000100:00080000:3.0:1789761938.988754:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761938.988760:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761938.988765:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761938.988780:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493996544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761938.988781:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761938.988836:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761938.988859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493996544:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761939.692563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761940.716595:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761941.740583:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761942.060765:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789761942.060814:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761942.060840:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761942.060880:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761942.060890:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789761942.060897:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789761942.060951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761938/real 1789761942] req@ffff8eb64514ad80 x1876699493996544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761942.060982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493996544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761942.060996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761942.061001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761942.061006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493996544:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761942.061015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761938/real 1789761942] req@ffff8eb645148700 x1876699493996288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761942.061031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493996288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761942.061041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761942.061046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761942.061050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789761942.061058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761938/real 1789761942] req@ffff8eb64514b480 x1876699493996416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761942.061078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493996416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761942.061087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761942.061091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761942.061094:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789761942.061105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761938/real 1789761942] req@ffff8eb645149180 x1876699493996160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761942.061118:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493996160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761942.061126:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761942.061130:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789761942.061133:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493996160:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789761942.764532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761943.788552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761944.108493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761944.108509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761944.108515:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761944.108529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1948 00000100:00080000:3.0:1789761944.108544:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761944.108571:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761944.108624:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699493996672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761944.108671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761944.108676:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761944.108679:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761944.108700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1948 00000100:00080000:3.0:1789761944.108705:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761944.108712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761944.108726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761944.108732:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493996672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761944.108747:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699493996800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761944.108762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761944.108766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761944.108768:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761944.108772:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1948 00000100:00080000:3.0:1789761944.108777:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761944.108782:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761944.108788:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761944.108802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699493996928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761944.108815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761944.108820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761944.108822:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761944.108826:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1948 00000800:00000100:0.0:1789761944.108832:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761944.108834:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761944.108842:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761944.108849:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761944.108863:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699493997056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761944.108895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761944/real 1789761944] req@ffff8eb645149180 x1876699493996672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761944.108917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699493996672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761944.108936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761944.108942:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761944.108947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493996672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761944.108980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761944.109009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761944.109023:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761944/real 1789761944] req@ffff8eb64514b480 x1876699493996800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761944.109046:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699493996800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761944.109055:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761944.109059:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761944.109063:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996800:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761944.109070:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996928:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761944.109094:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761944.109107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761944/real 1789761944] req@ffff8eb645148700 x1876699493996928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761944.109120:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699493996928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761944.109130:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761944.109134:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761944.109137:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493996928:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761944.109144:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493997056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761944.109167:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761944.109180:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761944/real 1789761944] req@ffff8eb64514ad80 x1876699493997056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761944.109192:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699493997056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761944.109207:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761944.109211:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761944.109214:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493997056:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761944.812599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761945.836514:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761946.860567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761947.884523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761948.908559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761949.228460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761949.228467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761949.228472:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761949.228479:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1953 00000100:00080000:3.0:1789761949.228488:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761949.228494:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761949.228527:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699493997184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761949.228556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761949.228560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761949.228563:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761949.228568:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1953 00000100:00080000:3.0:1789761949.228573:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761949.228578:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761949.228587:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761949.228601:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493997184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761949.228611:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699493997312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761949.228621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761949.228623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761949.228625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761949.228627:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1953 00000100:00080000:3.0:1789761949.228630:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761949.228633:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761949.228636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761949.228643:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699493997440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761949.228656:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761949.228660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761949.228662:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761949.228667:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1953 00000100:00080000:3.0:1789761949.228672:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761949.228677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761949.228684:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761949.228694:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761949.228706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699493997568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761949.228750:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761949/real 1789761949] req@ffff8eb645149880 x1876699493997184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.228766:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699493997184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.228775:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.228778:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761949.228781:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493997184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761949.228807:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761949.228824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761949.228832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761949/real 1789761949] req@ffff8eb64514b100 x1876699493997312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.228840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699493997312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.228851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.228853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761949.228856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997312:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761949.228859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997440:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761949.228872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761949.228878:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761949/real 1789761949] req@ffff8eb64514bb80 x1876699493997440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.228885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699493997440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.228891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.228893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761949.228894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761949.228898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493997568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761949.228908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761949.228915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761949/real 1789761949] req@ffff8eb645148000 x1876699493997568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761949.228922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699493997568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761949.228927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761949.228929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761949.228932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493997568:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761949.932615:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761950.956610:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761951.980569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761953.004569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761954.028526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761954.348484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761954.348497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761954.348503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761954.348516:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1958 00000100:00080000:3.0:1789761954.348529:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761954.348539:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761954.348586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699493997696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761954.348617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761954.348622:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761954.348624:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761954.348629:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1958 00000100:00080000:3.0:1789761954.348632:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761954.348637:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761954.348649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761954.348662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699493997824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761954.348674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761954.348679:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493997696:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761954.348680:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761954.348683:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761954.348692:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1958 00000100:00080000:3.0:1789761954.348695:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761954.348699:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761954.348705:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761954.348720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493997952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761954.348732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761954.348736:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761954.348738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761954.348742:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1958 00000100:00080000:3.0:1789761954.348749:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761954.348754:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761954.348759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761954.348764:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789761954.348782:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493998080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761954.348817:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761954/real 1789761954] req@ffff8eb64514b800 x1876699493997696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761954.348839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699493997696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761954.348852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761954.348857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761954.348861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493997696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761954.348888:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761954.348909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761954.348922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761954/real 1789761954] req@ffff8eb64514a300 x1876699493997824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761954.348935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699493997824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761954.348944:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761954.348947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761954.348951:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761954.348958:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997952:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761954.348975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761954.348984:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761954/real 1789761954] req@ffff8eb645148e00 x1876699493997952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761954.348997:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493997952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761954.349007:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761954.349010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761954.349013:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493997952:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761954.349019:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493998080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761954.349036:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761954.349045:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761954/real 1789761954] req@ffff8eb645148380 x1876699493998080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761954.349057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493998080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761954.349066:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761954.349069:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761954.349073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493998080:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761955.052593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761956.076537:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761957.100596:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761958.124553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761959.148589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761959.468569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761959.468581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761959.468585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761959.468596:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1963 00000100:00080000:1.0:1789761959.468608:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761959.468615:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761959.468662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699493998208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761959.468689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761959.468692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761959.468693:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761959.468697:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1963 00000100:00080000:1.0:1789761959.468699:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761959.468702:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761959.468711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761959.468721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699493998336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761959.468729:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761959.468735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761959.468736:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493998208:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761959.468737:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761959.468743:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1963 00000100:00080000:1.0:1789761959.468746:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761959.468749:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761959.468753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761959.468761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699493998464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761959.468768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761959.468771:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761959.468773:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761959.468776:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1963 00000100:00080000:1.0:1789761959.468778:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761959.468780:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761959.468783:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761959.468790:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699493998592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761959.468818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761959.468870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761959/real 1789761959] req@ffff8eb65016ed80 x1876699493998208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761959.468888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699493998208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761959.468897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761959.468900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761959.468903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493998208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761959.468928:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761959.468946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761959.468955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761959/real 1789761959] req@ffff8eb65016f480 x1876699493998336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761959.468963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699493998336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761959.468968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761959.468971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761959.468973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998336:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761959.468976:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998464:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761959.468987:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761959.468993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761959/real 1789761959] req@ffff8eb65016dc00 x1876699493998464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761959.469005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699493998464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761959.469010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761959.469012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761959.469014:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998464:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761959.469017:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493998592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761959.469027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761959.469033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761959/real 1789761959] req@ffff8eb65016df80 x1876699493998592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761959.469041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699493998592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761959.469045:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761959.469047:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761959.469049:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493998592:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761960.172534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761961.196575:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761962.220611:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761963.244556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761964.268614:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761964.588529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761964.588540:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761964.588545:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761964.588555:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1969 00000100:00080000:3.0:1789761964.588567:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761964.588575:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761964.588617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493998720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761964.588644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761964.588647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761964.588648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761964.588652:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1969 00000100:00080000:3.0:1789761964.588654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761964.588657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761964.588666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761964.588677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493998848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761964.588685:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761964.588687:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761964.588689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761964.588691:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1969 00000100:00080000:3.0:1789761964.588693:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761964.588699:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761964.588704:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761964.588712:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493998720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761964.588717:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493998976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761964.588732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761964.588737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761964.588739:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761964.588744:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1969 00000100:00080000:3.0:1789761964.588748:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761964.588753:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761964.588757:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761964.588776:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493999104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761964.588796:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761964.588843:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761964/real 1789761964] req@ffff8eb645149c00 x1876699493998720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761964.588871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493998720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761964.588883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761964.588887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761964.588892:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493998720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761964.588900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761964.588924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761964.588937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761964/real 1789761964] req@ffff8eb645149500 x1876699493998848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761964.588950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493998848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761964.588960:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761964.588963:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761964.588967:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998848:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761964.589007:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998976:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761964.589030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761964.589042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761964/real 1789761964] req@ffff8eb645148a80 x1876699493998976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761964.589066:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493998976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761964.589082:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761964.589086:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761964.589089:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493998976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761964.589096:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493999104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761964.589115:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761964.589126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761964/real 1789761964] req@ffff8eb64514a680 x1876699493999104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761964.589138:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493999104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761964.589148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761964.589151:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761964.589154:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493999104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761965.292527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761966.316582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761967.340508:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761968.364512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761969.388566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761969.708553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761969.708566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761969.708572:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761969.708585:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1974 00000100:00080000:3.0:1789761969.708615:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761969.708622:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761969.708670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699493999232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761969.708698:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761969.708701:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761969.708703:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761969.708706:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1974 00000100:00080000:3.0:1789761969.708709:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761969.708712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761969.708723:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761969.708738:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699493999360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761969.708752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761969.708753:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493999232:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789761969.708760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761969.708764:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761969.708769:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1974 00000100:00080000:3.0:1789761969.708774:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761969.708779:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761969.708786:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761969.708798:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699493999488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761969.708807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761969.708810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761969.708811:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761969.708815:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1974 00000100:00080000:3.0:1789761969.708827:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761969.708831:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761969.708852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789761969.708854:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761969.708866:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699493999616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761969.708902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761969/real 1789761969] req@ffff8eb64514a680 x1876699493999232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761969.708920:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699493999232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761969.708936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761969.708939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761969.708943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493999232:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761969.708966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761969.708983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761969.709011:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761969/real 1789761969] req@ffff8eb645148a80 x1876699493999360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761969.709031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699493999360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761969.709044:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761969.709049:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761969.709054:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761969.709064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999488:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761969.709097:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761969.709130:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761969/real 1789761969] req@ffff8eb645149500 x1876699493999488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761969.709188:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699493999488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761969.709195:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761969.709198:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761969.709200:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999488:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761969.709205:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493999616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761969.709222:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761969.709229:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761969/real 1789761969] req@ffff8eb645149c00 x1876699493999616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761969.709238:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699493999616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761969.709244:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761969.709246:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761969.709248:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699493999616:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761970.412511:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761971.436554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761972.460678:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761973.484571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761974.508507:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761974.828456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761974.828464:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761974.828467:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761974.828475:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1979 00000100:00080000:3.0:1789761974.828484:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761974.828490:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761974.828520:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699493999744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761974.828542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761974.828546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761974.828548:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761974.828552:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1979 00000100:00080000:3.0:1789761974.828556:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761974.828561:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761974.828571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761974.828586:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493999744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761974.828588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699493999872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761974.828602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761974.828605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761974.828606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761974.828610:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1979 00000100:00080000:3.0:1789761974.828613:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761974.828616:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761974.828621:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761974.828631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494000000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761974.828638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761974.828641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761974.828642:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761974.828645:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1979 00000800:00000100:0.0:1789761974.828648:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789761974.828649:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761974.828656:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761974.828660:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761974.828668:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494000128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761974.828688:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761974/real 1789761974] req@ffff8eb645148380 x1876699493999744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761974.828702:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699493999744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761974.828710:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761974.828713:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761974.828716:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699493999744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761974.828733:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761974.828748:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761974.828756:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761974/real 1789761974] req@ffff8eb645148e00 x1876699493999872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761974.828763:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699493999872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761974.828769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761974.828770:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761974.828772:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699493999872:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761974.828776:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000000:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761974.828787:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761974.828793:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761974/real 1789761974] req@ffff8eb64514a300 x1876699494000000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761974.828800:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494000000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761974.828805:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761974.828806:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761974.828808:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000000:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761974.828812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494000128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761974.828822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761974.828828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761974/real 1789761974] req@ffff8eb64514b800 x1876699494000128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761974.828839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494000128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761974.828844:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761974.828846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761974.828848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494000128:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761975.532528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761976.556553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761977.580565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761978.604535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761979.628591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761979.948452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761979.948461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761979.948464:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761979.948472:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1984 00000100:00080000:3.0:1789761979.948480:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761979.948486:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789761979.948516:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494000256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761979.948540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761979.948542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761979.948544:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761979.948546:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1984 00000100:00080000:3.0:1789761979.948549:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761979.948551:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761979.948559:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761979.948572:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494000384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761979.948580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761979.948583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761979.948584:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761979.948587:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1984 00000100:00080000:3.0:1789761979.948590:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761979.948593:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761979.948596:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761979.948602:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494000512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761979.948609:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761979.948611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761979.948613:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761979.948616:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1984 00000100:00080000:3.0:1789761979.948618:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761979.948621:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789761979.948625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761979.948636:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494000256:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789761979.948637:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494000640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761979.948738:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761979.948794:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761979/real 1789761979] req@ffff8eb645148000 x1876699494000256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761979.948816:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494000256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761979.948829:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761979.948834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761979.948838:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494000256:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761979.948847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761979.948873:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761979.948887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761979/real 1789761979] req@ffff8eb64514bb80 x1876699494000384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761979.948908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494000384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761979.948918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761979.948922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761979.948925:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761979.948933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000512:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761979.948955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761979.948966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761979/real 1789761979] req@ffff8eb64514b100 x1876699494000512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761979.948979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494000512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761979.948988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761979.948992:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761979.948995:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761979.949030:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494000640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761979.949051:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761979.949063:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761979/real 1789761979] req@ffff8eb645149880 x1876699494000640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761979.949093:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494000640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761979.949109:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761979.949113:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761979.949116:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494000640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761980.652544:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761981.676589:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761982.700539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761983.724542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761984.748528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761985.068509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761985.068520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761985.068525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761985.068535:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1989 00000100:00080000:1.0:1789761985.068547:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761985.068554:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761985.068592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494000768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761985.068619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761985.068626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761985.068628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761985.068633:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1989 00000100:00080000:1.0:1789761985.068639:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761985.068644:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761985.068654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761985.068670:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494000768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761985.068683:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494000896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761985.068699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761985.068703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761985.068705:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761985.068711:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1989 00000100:00080000:1.0:1789761985.068716:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761985.068720:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761985.068725:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761985.068729:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761985.068748:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494001024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761985.068758:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761985/real 1789761985] req@ffff8eb65016c700 x1876699494000768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761985.068764:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761985.068772:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761985.068774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761985.068777:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494000768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761985.068783:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1989 00000100:00080000:1.0:1789761985.068789:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761985.068797:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761985.068797:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:0.0:1789761985.068806:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000008:00100000:1.0:1789761985.068808:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761985.068810:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494000768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761985.068828:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494001152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761985.068834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761985.068859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761985.068870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761985/real 1789761985] req@ffff8eb65016ce00 x1876699494000896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761985.068879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494000896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761985.068889:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761985.068891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761985.068894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494000896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761985.068898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001024:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761985.068909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761985.068915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761985/real 1789761985] req@ffff8eb65016c380 x1876699494001024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761985.068922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494001024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761985.068927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761985.068929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761985.068931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001024:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761985.068940:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494001152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761985.068950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761985.068957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761985/real 1789761985] req@ffff8eb65016fb80 x1876699494001152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761985.068964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494001152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761985.068969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761985.068970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761985.068973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494001152:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789761985.772547:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761986.796537:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761987.820540:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761988.844543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761989.868571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761990.188459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761990.188470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761990.188475:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761990.188486:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1994 00000100:00080000:1.0:1789761990.188498:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761990.188505:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761990.188546:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494001280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761990.188570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761990.188574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761990.188576:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761990.188582:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1994 00000100:00080000:1.0:1789761990.188586:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761990.188592:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761990.188605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761990.188621:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494001280:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789761990.188622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494001408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761990.188641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761990.188645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761990.188647:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761990.188651:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1994 00000100:00080000:1.0:1789761990.188656:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761990.188661:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761990.188667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761990.188682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494001536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761990.188695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761990.188699:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761990.188701:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761990.188705:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1994 00000100:00080000:1.0:1789761990.188710:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761990.188715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789761990.188719:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761990.188727:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761990.188741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494001664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761990.188774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761990/real 1789761990] req@ffff8eb65016d500 x1876699494001280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761990.188796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494001280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761990.188808:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761990.188813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761990.188819:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494001280:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761990.188848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761990.188876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761990.188890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761990/real 1789761990] req@ffff8eb65016e300 x1876699494001408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761990.188904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494001408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761990.188913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761990.188917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761990.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761990.188931:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001536:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761990.188955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761990.188979:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761990/real 1789761990] req@ffff8eb65016f800 x1876699494001536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761990.189010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494001536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761990.189020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761990.189024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761990.189028:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761990.189044:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494001664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761990.189067:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761990.189080:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761990/real 1789761990] req@ffff8eb65016e680 x1876699494001664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761990.189092:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494001664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761990.189102:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761990.189106:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761990.189109:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494001664:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761990.892510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761991.916544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761992.940551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761993.964581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761994.988567:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761995.308540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761995.308551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761995.308556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761995.308568:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 1999 00000100:00080000:1.0:1789761995.308580:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761995.308587:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789761995.308636:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494001792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761995.308667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761995.308674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761995.308677:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761995.308682:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1999 00000100:00080000:1.0:1789761995.308686:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761995.308691:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761995.308703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761995.308716:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494001920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761995.308726:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494001792:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789761995.308727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761995.308735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761995.308739:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761995.308743:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 1999 00000100:00080000:1.0:1789761995.308747:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761995.308751:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789761995.308757:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761995.308773:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494002048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761995.308785:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761995.308790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761995.308792:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761995.308797:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 1999 00000100:00080000:1.0:1789761995.308802:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761995.308816:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789761995.308817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789761995.308822:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761995.308834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494002176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761995.308871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761995/real 1789761995] req@ffff8eb65016ea00 x1876699494001792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761995.308893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494001792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761995.308906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761995.308911:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761995.308915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494001792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789761995.308944:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761995.309028:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761995.309042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761995/real 1789761995] req@ffff8eb65016d880 x1876699494001920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761995.309056:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494001920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761995.309066:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761995.309069:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761995.309073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494001920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789761995.309080:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002048:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789761995.309102:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761995.309113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761995/real 1789761995] req@ffff8eb65016e680 x1876699494002048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761995.309133:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494002048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761995.309142:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761995.309145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761995.309149:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002048:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789761995.309155:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494002176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789761995.309175:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789761995.309186:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761995/real 1789761995] req@ffff8eb65016f800 x1876699494002176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761995.309199:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494002176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761995.309208:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761995.309211:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789761995.309215:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494002176:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789761996.012635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761997.036610:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761998.060570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761999.084632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762000.108586:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762000.428515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762000.428530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762000.428535:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762000.428549:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2004 00000100:00080000:2.0:1789762000.428563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762000.428572:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762000.428625:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494002304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762000.428654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762000.428657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762000.428658:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762000.428662:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2004 00000100:00080000:2.0:1789762000.428665:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762000.428669:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762000.428678:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762000.428690:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494002432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762000.428700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762000.428704:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762000.428705:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494002304:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762000.428706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762000.428713:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2004 00000100:00080000:2.0:1789762000.428716:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762000.428719:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762000.428722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762000.428731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494002560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762000.428738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762000.428741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762000.428743:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762000.428746:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2004 00000100:00080000:2.0:1789762000.428748:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762000.428751:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762000.428754:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762000.428761:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494002688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762000.428781:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762000.428824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762000/real 1789762000] req@ffff8eb650f80700 x1876699494002304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762000.428844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494002304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762000.428853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762000.428856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762000.428859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494002304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762000.428881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762000.428896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762000.428904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762000/real 1789762000] req@ffff8eb650f80e00 x1876699494002432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762000.428912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494002432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762000.428931:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762000.428934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762000.428936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762000.428940:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002560:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762000.428957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762000.428965:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762000/real 1789762000] req@ffff8eb650f83480 x1876699494002560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762000.428973:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494002560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762000.428983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762000.428985:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762000.428988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002560:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762000.428991:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494002688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762000.429002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762000.429012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762000/real 1789762000] req@ffff8eb650f82a00 x1876699494002688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762000.429023:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494002688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762000.429032:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762000.429035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762000.429039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494002688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762001.132552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762002.156556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762003.180626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762004.204535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762005.228610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762005.548545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762005.548559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762005.548566:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762005.548581:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2010 00000100:00080000:3.0:1789762005.548596:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762005.548607:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762005.548659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494002816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762005.548695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762005.548699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762005.548700:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762005.548705:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2010 00000100:00080000:3.0:1789762005.548709:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762005.548714:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762005.548726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762005.548740:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494002944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762005.548753:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762005.548759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494002816:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762005.548759:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762005.548762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762005.548772:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2010 00000100:00080000:3.0:1789762005.548776:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762005.548780:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762005.548786:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762005.548799:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494003072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762005.548811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762005.548815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762005.548817:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762005.548823:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2010 00000100:00080000:3.0:1789762005.548827:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762005.548831:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762005.548837:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762005.548849:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494003200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762005.548914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762005.548950:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762005.548977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494003200:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762006.252569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762007.276568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762008.300628:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762008.620736:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762008.620784:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762008.620805:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762008.620867:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762008.620878:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762008.620892:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762008.620989:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762005/real 1789762008] req@ffff8eb645148700 x1876699494002944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762008.621033:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494002944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762008.621049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762008.621055:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762008.621059:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494002944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762008.621068:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762005/real 1789762008] req@ffff8eb64514b480 x1876699494003072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762008.621083:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494003072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762008.621093:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762008.621097:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762008.621101:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762008.621110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762005/real 1789762008] req@ffff8eb645149180 x1876699494003200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762008.621123:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494003200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762008.621131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762008.621136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762008.621139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494003200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762008.621147:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762005/real 1789762008] req@ffff8eb64514ad80 x1876699494002816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762008.621161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494002816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762008.621170:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762008.621174:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762008.621178:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494002816:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762009.324621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762010.348559:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762010.668558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762010.668569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762010.668572:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762010.668580:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2015 00000100:00080000:1.0:1789762010.668589:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762010.668595:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762010.668624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494003328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762010.668645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762010.668648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762010.668649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762010.668652:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2015 00000100:00080000:1.0:1789762010.668654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762010.668657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762010.668664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762010.668675:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494003456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762010.668686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762010.668689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762010.668692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762010.668696:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2015 00000100:00080000:1.0:1789762010.668700:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762010.668705:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762010.668710:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762010.668722:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494003584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762010.668734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762010.668741:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494003328:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762010.668741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762010.668749:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762010.668757:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2015 00000100:00080000:1.0:1789762010.668761:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762010.668767:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762010.668774:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762010.668789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494003712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762010.668859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762010.668920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762010/real 1789762010] req@ffff8eb65016e300 x1876699494003328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762010.668943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494003328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762010.668956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762010.668962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762010.668966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494003328:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762010.668975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762010.669003:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762010.669017:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762010/real 1789762010] req@ffff8eb65016d500 x1876699494003456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762010.669029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494003456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762010.669039:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762010.669043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762010.669046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003456:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762010.669079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003584:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762010.669103:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762010.669116:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762010/real 1789762010] req@ffff8eb65016fb80 x1876699494003584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762010.669129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494003584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762010.669139:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762010.669143:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762010.669146:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003584:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762010.669153:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494003712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762010.669188:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762010.669201:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762010/real 1789762010] req@ffff8eb65016c380 x1876699494003712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762010.669214:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494003712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762010.669222:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762010.669226:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762010.669230:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494003712:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762011.372559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762012.396573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762013.420601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762014.444658:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762015.468619:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762015.788507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762015.788516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762015.788519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762015.788527:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2020 00000100:00080000:3.0:1789762015.788537:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762015.788543:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762015.788580:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494003840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762015.788613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762015.788618:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762015.788620:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762015.788625:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2020 00000100:00080000:3.0:1789762015.788629:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762015.788633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762015.788643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762015.788658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494003968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762015.788671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762015.788676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762015.788678:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494003840:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762015.788678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762015.788686:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2020 00000100:00080000:3.0:1789762015.788691:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762015.788694:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762015.788699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762015.788710:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494004096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762015.788719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762015.788721:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762015.788722:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762015.788725:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2020 00000100:00080000:3.0:1789762015.788728:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762015.788730:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762015.788733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762015.788741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494004224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762015.788754:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762015.788792:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762015/real 1789762015] req@ffff8eb64514ad80 x1876699494003840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762015.788825:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494003840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762015.788834:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762015.788837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762015.788840:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494003840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762015.788859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762015.788876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762015.788885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762015/real 1789762015] req@ffff8eb645149180 x1876699494003968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762015.788893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494003968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762015.788898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762015.788901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762015.788903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494003968:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762015.788907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004096:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762015.788918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762015.788924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762015/real 1789762015] req@ffff8eb64514b480 x1876699494004096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762015.788931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494004096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762015.788936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762015.788938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762015.788940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762015.788943:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494004224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762015.788953:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762015.788959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762015/real 1789762015] req@ffff8eb645148700 x1876699494004224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762015.788966:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494004224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762015.788972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762015.788973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762015.788975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494004224:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762016.492557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762017.516590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762018.540644:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762019.564607:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762020.588641:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762020.908573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762020.908585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762020.908589:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762020.908601:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2025 00000100:00080000:1.0:1789762020.908613:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762020.908620:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762020.908665:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494004352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762020.908693:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762020.908696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762020.908697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762020.908701:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2025 00000100:00080000:1.0:1789762020.908704:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762020.908706:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762020.908722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762020.908732:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494004480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762020.908740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762020.908743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762020.908745:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762020.908747:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2025 00000100:00080000:1.0:1789762020.908750:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762020.908755:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762020.908759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494004352:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762020.908775:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762020.908788:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494004608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762020.908801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762020.908804:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762020.908805:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762020.908809:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2025 00000100:00080000:1.0:1789762020.908813:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762020.908816:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762020.908819:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762020.908827:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494004736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762020.908860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762020.908913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762020/real 1789762020] req@ffff8eb65016ce00 x1876699494004352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762020.908936:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494004352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762020.908949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762020.908954:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762020.908959:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494004352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762020.908969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762020.908993:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762020.909007:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762020/real 1789762020] req@ffff8eb65016c700 x1876699494004480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762020.909022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494004480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762020.909031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762020.909036:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762020.909040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762020.909072:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004608:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762020.909094:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762020.909105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762020/real 1789762020] req@ffff8eb65016df80 x1876699494004608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762020.909126:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494004608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762020.909135:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762020.909139:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762020.909144:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004608:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762020.909151:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494004736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762020.909169:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762020.909181:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762020/real 1789762020] req@ffff8eb65016dc00 x1876699494004736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762020.909194:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494004736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762020.909202:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762020.909205:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762020.909209:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494004736:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762021.612558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762022.636621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762023.660572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762024.684596:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762025.708648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762026.028519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762026.028530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762026.028534:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762026.028547:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2030 00000100:00080000:1.0:1789762026.028558:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762026.028566:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762026.028610:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494004864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762026.028636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762026.028638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762026.028640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762026.028644:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2030 00000100:00080000:1.0:1789762026.028646:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762026.028649:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762026.028659:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762026.028670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494004992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762026.028679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762026.028682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762026.028683:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762026.028688:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2030 00000100:00100000:0.0:1789762026.028690:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494004864:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762026.028691:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762026.028698:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762026.028702:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762026.028711:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494005120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762026.028718:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762026.028734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762026.028737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762026.028740:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2030 00000100:00080000:1.0:1789762026.028744:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762026.028747:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762026.028752:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762026.028761:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494005248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762026.028788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762026.028844:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762026/real 1789762026] req@ffff8eb65016f480 x1876699494004864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762026.028871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494004864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762026.028880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762026.028884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762026.028887:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494004864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762026.028911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762026.028931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762026.028941:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762026/real 1789762026] req@ffff8eb65016ed80 x1876699494004992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762026.028950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494004992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762026.028956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762026.028958:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762026.028960:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494004992:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762026.028964:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005120:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762026.028976:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762026.028982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762026/real 1789762026] req@ffff8eb65016d180 x1876699494005120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762026.028989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494005120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762026.029000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762026.029002:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762026.029004:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005120:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762026.029007:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494005248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762026.029018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762026.029025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762026/real 1789762026] req@ffff8eb65016c000 x1876699494005248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762026.029032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494005248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762026.029037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762026.029039:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762026.029041:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494005248:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762026.732599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762027.756617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762028.780593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762029.804589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762030.828684:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762031.148504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762031.148517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762031.148524:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762031.148556:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2035 00000100:00080000:1.0:1789762031.148569:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762031.148578:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762031.148629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494005376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762031.148658:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762031.148662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762031.148663:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762031.148668:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2035 00000100:00080000:1.0:1789762031.148671:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762031.148676:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762031.148705:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762031.148709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494005376:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762031.148745:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494005504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762031.148764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762031.148770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762031.148773:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762031.148777:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2035 00000100:00080000:1.0:1789762031.148793:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762031.148799:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762031.148806:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762031.148812:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762031.148823:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494005632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762031.148841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762031.148844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762031.148845:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762031.148849:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2035 00000100:00080000:1.0:1789762031.148852:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762031.148856:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762031.148861:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762031.148867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762031/real 1789762031] req@ffff8eb65016ca80 x1876699494005376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762031.148878:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494005760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762031.148895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494005376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762031.148921:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762031.148926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762031.148932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494005376:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762031.148980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762031.149041:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762031.149059:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762031/real 1789762031] req@ffff8eb65016f100 x1876699494005504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762031.149092:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494005504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762031.149100:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762031.149103:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762031.149106:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762031.149112:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005632:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762031.149141:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762031.149150:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762031/real 1789762031] req@ffff8eb65016c000 x1876699494005632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762031.149160:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494005632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762031.149167:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762031.149170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762031.149172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494005632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762031.149178:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494005760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762031.149197:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762031.149210:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762031/real 1789762031] req@ffff8eb65016d180 x1876699494005760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762031.149227:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494005760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762031.149237:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762031.149242:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762031.149246:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494005760:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762031.852548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762032.876560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762033.900567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762034.924552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762035.948606:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762036.268506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762036.268517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762036.268521:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762036.268533:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2040 00000100:00080000:0.0:1789762036.268545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762036.268552:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762036.268597:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494005888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762036.268625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762036.268628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762036.268630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762036.268633:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2040 00000100:00080000:0.0:1789762036.268636:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762036.268639:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762036.268659:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762036.268670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494006016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762036.268679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762036.268682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762036.268684:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762036.268687:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2040 00000100:00080000:0.0:1789762036.268690:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762036.268692:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762036.268696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762036.268704:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494006144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762036.268713:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762036.268717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762036.268722:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762036.268724:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494005888:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762036.268727:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2040 00000100:00080000:0.0:1789762036.268738:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762036.268743:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762036.268745:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762036.268753:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494006272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762036.268800:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762036.268835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762036/real 1789762036] req@ffff8eb64304f800 x1876699494005888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762036.268849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494005888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762036.268858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762036.268861:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762036.268864:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494005888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762036.268870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762036.268884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762036.268891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762036/real 1789762036] req@ffff8eb64304c380 x1876699494006016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762036.268899:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494006016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762036.268905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762036.268907:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762036.268909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762036.268927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006144:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762036.268939:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762036.268945:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762036/real 1789762036] req@ffff8eb64304dc00 x1876699494006144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762036.268952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494006144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762036.268957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762036.268959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762036.268961:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006144:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762036.268966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494006272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762036.268976:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762036.268982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762036/real 1789762036] req@ffff8eb64304ce00 x1876699494006272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762036.268989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494006272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762036.268994:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762036.268996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762036.268998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494006272:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762036.972567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762037.996749:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762039.020581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762040.044646:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762041.068558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762041.388535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762041.388545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762041.388549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762041.388559:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2045 00000100:00080000:3.0:1789762041.388570:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762041.388577:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762041.388635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494006400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762041.388672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762041.388675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762041.388677:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762041.388682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2045 00000100:00080000:3.0:1789762041.388685:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762041.388689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762041.388699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762041.388710:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494006528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762041.388720:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494006400:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762041.388720:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762041.388725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762041.388728:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762041.388734:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2045 00000100:00080000:3.0:1789762041.388736:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762041.388738:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762041.388741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762041.388748:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494006656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762041.388755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762041.388758:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762041.388759:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762041.388762:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2045 00000100:00080000:3.0:1789762041.388765:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762041.388767:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762041.388770:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762041.388777:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494006784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762041.388813:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762041.388878:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762041/real 1789762041] req@ffff8eb645149880 x1876699494006400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762041.388900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494006400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762041.388922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762041.388926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762041.388931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494006400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762041.388961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762041.388999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762041.389014:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762041/real 1789762041] req@ffff8eb64514b100 x1876699494006528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762041.389027:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494006528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762041.389037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762041.389040:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762041.389044:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006528:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762041.389051:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006656:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762041.389073:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762041.389084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762041/real 1789762041] req@ffff8eb64514bb80 x1876699494006656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762041.389097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494006656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762041.389106:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762041.389109:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762041.389113:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494006656:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762041.389119:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494006784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762041.389151:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762041.389163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762041/real 1789762041] req@ffff8eb645148000 x1876699494006784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762041.389175:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494006784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762041.389184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762041.389187:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762041.389191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494006784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762042.092645:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762043.116535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762044.140579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762045.164667:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762046.188664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762046.508488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762046.508497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762046.508501:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762046.508512:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2051 00000100:00080000:3.0:1789762046.508522:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762046.508529:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762046.508567:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494006912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762046.508612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762046.508617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762046.508620:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762046.508625:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2051 00000100:00080000:3.0:1789762046.508630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762046.508635:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762046.508644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762046.508648:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494006912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762046.508659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494007040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762046.508669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762046.508672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762046.508673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762046.508677:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2051 00000100:00080000:3.0:1789762046.508680:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762046.508682:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762046.508686:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762046.508694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494007168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762046.508701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762046.508703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762046.508704:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762046.508707:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2051 00000100:00080000:3.0:1789762046.508709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762046.508715:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762046.508717:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762046.508719:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762046.508728:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494007296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762046.508755:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762046/real 1789762046] req@ffff8eb64514b800 x1876699494006912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762046.508771:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494006912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762046.508779:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762046.508782:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762046.508785:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494006912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762046.508802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762046.508817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762046.508825:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762046/real 1789762046] req@ffff8eb64514a300 x1876699494007040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762046.508833:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494007040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762046.508838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762046.508840:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762046.508843:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762046.508846:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007168:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762046.508856:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762046.508862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762046/real 1789762046] req@ffff8eb645148e00 x1876699494007168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762046.508869:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494007168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762046.508878:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762046.508881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762046.508883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762046.508886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494007296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762046.508896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762046.508901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762046/real 1789762046] req@ffff8eb645148380 x1876699494007296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762046.508908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494007296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762046.508913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762046.508915:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762046.508917:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494007296:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762047.212631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762048.236634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762049.260509:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762050.284532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762051.308522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762051.628467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762051.628477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762051.628483:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762051.628495:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2056 00000100:00080000:3.0:1789762051.628513:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762051.628522:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762051.628592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494007424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762051.628621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762051.628626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762051.628628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762051.628633:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2056 00000100:00080000:3.0:1789762051.628637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762051.628642:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762051.628653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762051.628660:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494007424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762051.628670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494007552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762051.628684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762051.628689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762051.628691:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762051.628695:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2056 00000100:00080000:3.0:1789762051.628700:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762051.628704:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762051.628710:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762051.628715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762051.628729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494007680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762051.628742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762051.628747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762051.628751:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762051.628752:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762051/real 1789762051] req@ffff8eb645149c00 x1876699494007424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762051.628761:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2056 00000100:00080000:3.0:1789762051.628768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762051.628772:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762051.628776:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494007424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762051.628779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762051.628790:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762051.628795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494007808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762051.628797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762051.628805:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494007424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762051.628830:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762051.628845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762051.628854:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762051/real 1789762051] req@ffff8eb645149500 x1876699494007552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762051.628862:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494007552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762051.628867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762051.628870:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762051.628872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007552:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762051.628875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007680:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762051.628886:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762051.628892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762051/real 1789762051] req@ffff8eb645148a80 x1876699494007680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762051.628899:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494007680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762051.628904:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762051.628906:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762051.628908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494007680:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762051.628911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494007808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762051.628921:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762051.628927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762051/real 1789762051] req@ffff8eb64514a680 x1876699494007808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762051.628934:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494007808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762051.628939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762051.628941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762051.628943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494007808:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762052.332506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762053.356542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762054.380557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762055.404577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762056.428595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762056.748533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762056.748546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762056.748551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762056.748564:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2061 00000100:00080000:3.0:1789762056.748579:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762056.748587:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762056.748636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494007936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762056.748671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762056.748675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762056.748677:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762056.748682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2061 00000100:00080000:3.0:1789762056.748685:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762056.748689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762056.748702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762056.748716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494008064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762056.748727:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494007936:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762056.748731:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762056.748737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762056.748739:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762056.748744:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2061 00000100:00080000:3.0:1789762056.748747:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762056.748751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762056.748757:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762056.748770:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494008192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762056.748782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762056.748787:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762056.748789:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762056.748793:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2061 00000100:00080000:3.0:1789762056.748804:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762056.748810:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762056.748815:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762056.748817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762056.748831:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494008320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762056.748891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762056/real 1789762056] req@ffff8eb64514a680 x1876699494007936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762056.748908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494007936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762056.748917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762056.748921:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762056.748924:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494007936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762056.748947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762056.748967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762056.748977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762056/real 1789762056] req@ffff8eb645148a80 x1876699494008064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762056.748991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494008064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762056.748997:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762056.748999:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762056.749002:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008064:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762056.749005:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008192:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762056.749018:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762056.749024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762056/real 1789762056] req@ffff8eb645149500 x1876699494008192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762056.749031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494008192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762056.749037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762056.749038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762056.749040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762056.749051:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494008320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762056.749063:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762056.749070:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762056/real 1789762056] req@ffff8eb645149c00 x1876699494008320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762056.749080:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494008320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762056.749095:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762056.749099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762056.749102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494008320:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762057.452566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762058.476520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762059.500623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762060.524566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762061.548638:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762061.868516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762061.868528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762061.868533:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762061.868543:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2066 00000100:00080000:3.0:1789762061.868555:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762061.868562:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762061.868604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494008448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762061.868632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762061.868636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762061.868638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762061.868643:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2066 00000100:00080000:3.0:1789762061.868646:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762061.868650:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762061.868660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762061.868674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494008576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762061.868684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762061.868688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762061.868690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762061.868695:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2066 00000100:00080000:3.0:1789762061.868698:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762061.868707:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494008448:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762061.868707:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762061.868729:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762061.868749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494008704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762061.868763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762061.868766:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762061.868768:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762061.868775:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2066 00000100:00080000:3.0:1789762061.868779:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762061.868782:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762061.868787:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762061.868799:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494008832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762061.868805:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762061.868864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762061/real 1789762061] req@ffff8eb645148380 x1876699494008448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762061.868886:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494008448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762061.868898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762061.868903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762061.868908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494008448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762061.868917:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762061.868940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762061.868952:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762061/real 1789762061] req@ffff8eb645148e00 x1876699494008576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762061.868965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494008576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762061.868973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762061.868978:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762061.868982:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762061.869010:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008704:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762061.869030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762061.869040:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762061/real 1789762061] req@ffff8eb64514a300 x1876699494008704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762061.869053:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494008704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762061.869062:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762061.869065:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762061.869069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494008704:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762061.869075:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494008832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762061.869092:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762061.869102:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762061/real 1789762061] req@ffff8eb64514b800 x1876699494008832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762061.869114:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494008832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762061.869122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762061.869125:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762061.869129:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494008832:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762062.572641:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762063.596625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762064.620647:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762065.644642:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762066.668626:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762066.988552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762066.988564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762066.988568:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762066.988579:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2071 00000100:00080000:1.0:1789762066.988591:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762066.988599:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762066.988644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494008960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762066.988672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762066.988675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762066.988676:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762066.988680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2071 00000100:00080000:1.0:1789762066.988682:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762066.988685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762066.988695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762066.988706:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494009088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762066.988718:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762066.988722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762066.988725:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762066.988729:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2071 00000100:00080000:1.0:1789762066.988734:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762066.988736:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494008960:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762066.988741:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762066.988750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762066.988764:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494009216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762066.988773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762066.988776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762066.988778:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762066.988781:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2071 00000100:00080000:1.0:1789762066.988784:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762066.988787:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762066.988791:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762066.988798:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494009344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762066.988824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762066.988885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762066/real 1789762066] req@ffff8eb65016ed80 x1876699494008960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762066.988907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494008960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762066.988920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762066.988925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762066.988930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494008960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762066.988938:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762066.988961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762066.988976:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762066/real 1789762066] req@ffff8eb65016f480 x1876699494009088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762066.988990:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494009088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762066.988999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762066.989003:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762066.989007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009088:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762066.989038:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009216:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762066.989055:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762066.989063:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762066/real 1789762066] req@ffff8eb65016dc00 x1876699494009216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762066.989077:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494009216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762066.989083:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762066.989085:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762066.989087:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009216:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762066.989090:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494009344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762066.989101:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762066.989107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762066/real 1789762066] req@ffff8eb65016df80 x1876699494009344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762066.989115:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494009344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762066.989120:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762066.989121:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762066.989123:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494009344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762067.692647:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762068.716578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762069.740619:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762070.764573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762071.788583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762072.108528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762072.108537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762072.108540:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762072.108548:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2076 00000100:00080000:3.0:1789762072.108557:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762072.108564:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762072.108603:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494009472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762072.108635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762072.108639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762072.108640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762072.108644:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2076 00000100:00080000:3.0:1789762072.108647:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762072.108650:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762072.108658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762072.108670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494009600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762072.108678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762072.108681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762072.108682:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762072.108685:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2076 00000100:00080000:3.0:1789762072.108690:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762072.108692:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762072.108696:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762072.108705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494009728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762072.108716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762072.108720:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762072.108724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762072.108726:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494009472:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762072.108729:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2076 00000100:00080000:3.0:1789762072.108738:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762072.108743:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762072.108747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762072.108757:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494009856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762072.108874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762072.108927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762072.108969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494009856:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762072.812576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762073.836656:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762074.860599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762075.180653:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762075.180702:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762075.180720:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762075.180767:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762075.180778:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762075.180789:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762075.180824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762072/real 1789762075] req@ffff8eb64514b100 x1876699494009728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.180855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494009728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.180870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.180876:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762075.180881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762075.180891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762072/real 1789762075] req@ffff8eb645149880 x1876699494009856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.180906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494009856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.180916:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.180920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762075.180924:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494009856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762075.180931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762072/real 1789762075] req@ffff8eb645148000 x1876699494009472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.180945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494009472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.180954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.180957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762075.180962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494009472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762075.180970:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762072/real 1789762075] req@ffff8eb64514bb80 x1876699494009600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.180982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494009600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.180991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.180995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762075.180999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494009600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762075.884591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762076.908602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762077.228520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762077.228530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762077.228535:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762077.228547:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2081 00000100:00080000:0.0:1789762077.228560:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762077.228567:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762077.228611:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494009984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762077.228643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762077.228647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762077.228649:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762077.228654:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2081 00000100:00080000:0.0:1789762077.228658:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762077.228663:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762077.228677:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762077.228692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494010112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762077.228701:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762077.228704:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762077.228706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762077.228709:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2081 00000100:00080000:0.0:1789762077.228712:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762077.228715:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762077.228719:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762077.228731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494010240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762077.228738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762077.228741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762077.228742:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762077.228744:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2081 00000100:00080000:0.0:1789762077.228746:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762077.228749:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762077.228751:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762077.228757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494010368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762077.228859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494009984:192.168.203.160@tcp:250:kworker.0 00000800:00000100:3.0:1789762077.228933:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762077.228972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762077/real 1789762077] req@ffff8eb64304d500 x1876699494009984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762077.228992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494009984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762077.229004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762077.229008:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762077.229013:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494009984:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762077.229022:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762077.229046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762077.229060:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762077/real 1789762077] req@ffff8eb64304d880 x1876699494010112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762077.229079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494010112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762077.229087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762077.229091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762077.229095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010112:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762077.229107:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010240:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762077.229128:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762077.229138:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762077/real 1789762077] req@ffff8eb64304f480 x1876699494010240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762077.229149:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494010240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762077.229157:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762077.229160:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762077.229163:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010240:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762077.229169:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494010368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762077.229186:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762077.229195:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762077/real 1789762077] req@ffff8eb64304e680 x1876699494010368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762077.229205:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494010368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762077.229212:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762077.229215:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762077.229218:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494010368:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762077.932542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762078.956679:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762079.980590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762081.004625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762082.028559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762082.348473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762082.348493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762082.348497:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762082.348505:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2086 00000100:00080000:0.0:1789762082.348515:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762082.348521:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762082.348565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494010496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762082.348586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762082.348589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762082.348590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762082.348593:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2086 00000100:00080000:0.0:1789762082.348595:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762082.348598:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762082.348608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762082.348632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494010624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762082.348645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789762082.348647:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494010496:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762082.348653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762082.348657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762082.348662:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2086 00000100:00080000:0.0:1789762082.348665:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762082.348668:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762082.348673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762082.348684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494010752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762082.348693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762082.348695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762082.348697:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762082.348699:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2086 00000100:00080000:0.0:1789762082.348702:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762082.348706:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762082.348710:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762082.348714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762082.348722:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494010880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762082.348748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762082/real 1789762082] req@ffff8eb64304ca80 x1876699494010496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762082.348762:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494010496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762082.348770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762082.348773:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762082.348776:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494010496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762082.348797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762082.348819:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762082.348831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762082/real 1789762082] req@ffff8eb64304ea00 x1876699494010624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762082.348844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494010624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762082.348850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762082.348852:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762082.348855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010624:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762082.348859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010752:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762082.348873:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762082.348881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762082/real 1789762082] req@ffff8eb64304c700 x1876699494010752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762082.348888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494010752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762082.348893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762082.348895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762082.348897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494010752:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762082.348900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494010880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762082.348910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762082.348915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762082/real 1789762082] req@ffff8eb64304f100 x1876699494010880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762082.348923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494010880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762082.348928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762082.348929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762082.348931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494010880:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762083.052580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762084.076666:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762085.100623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762086.124568:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762087.148641:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762087.468519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762087.468535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762087.468552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762087.468568:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2091 00000100:00080000:0.0:1789762087.468584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762087.468596:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762087.468668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494011008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762087.468708:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762087.468713:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762087.468716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762087.468722:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2091 00000100:00080000:0.0:1789762087.468729:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762087.468736:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762087.468751:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762087.468771:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494011136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762087.468789:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789762087.468791:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494011008:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762087.468795:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762087.468802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762087.468813:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2091 00000100:00080000:0.0:1789762087.468817:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762087.468822:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762087.468830:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762087.468848:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494011264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762087.468863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762087.468868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762087.468872:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762087.468879:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2091 00000800:00000100:3.0:1789762087.468888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762087.468890:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762087.468899:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762087.468907:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762087.468927:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494011392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762087.468974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762087/real 1789762087] req@ffff8eb64304f100 x1876699494011008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762087.468994:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494011008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762087.469004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762087.469008:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762087.469012:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494011008:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762087.469036:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762087.469058:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762087.469068:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762087/real 1789762087] req@ffff8eb64304c700 x1876699494011136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762087.469078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494011136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762087.469085:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762087.469087:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762087.469090:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011136:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762087.469095:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011264:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762087.469109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762087.469116:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762087/real 1789762087] req@ffff8eb64304ea00 x1876699494011264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762087.469125:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494011264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762087.469144:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762087.469147:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762087.469150:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011264:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762087.469156:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494011392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762087.469169:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762087.469177:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762087/real 1789762087] req@ffff8eb64304ca80 x1876699494011392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762087.469186:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494011392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762087.469192:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762087.469194:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762087.469197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494011392:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762088.172640:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762089.196590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762090.220583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762091.244655:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762092.268533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762092.588591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762092.588605:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762092.588611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762092.588623:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2097 00000100:00080000:1.0:1789762092.588653:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762092.588663:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762092.588711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494011520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762092.588740:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762092.588744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762092.588746:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762092.588750:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2097 00000100:00080000:1.0:1789762092.588753:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762092.588757:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762092.588768:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762092.588782:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494011648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762092.588798:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762092.588819:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762092.588822:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762092.588829:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2097 00000100:00080000:1.0:1789762092.588833:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762092.588843:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762092.588847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494011520:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762092.588852:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762092.588888:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494011776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762092.588898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762092.588901:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762092.588902:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762092.588913:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2097 00000100:00080000:1.0:1789762092.588917:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762092.588921:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762092.588925:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762092.588934:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494011904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762092.588944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762092.588991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762092/real 1789762092] req@ffff8eb65016c700 x1876699494011520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762092.589005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494011520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762092.589018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762092.589021:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762092.589024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494011520:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762092.589030:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762092.589047:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762092.589056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762092/real 1789762092] req@ffff8eb65016ce00 x1876699494011648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762092.589064:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494011648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762092.589069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762092.589071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762092.589073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011648:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762092.589092:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011776:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762092.589103:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762092.589109:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762092/real 1789762092] req@ffff8eb65016c380 x1876699494011776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762092.589117:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494011776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762092.589122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762092.589124:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762092.589126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494011776:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762092.589130:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494011904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762092.589163:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762092.589170:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762092/real 1789762092] req@ffff8eb65016fb80 x1876699494011904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762092.589178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494011904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762092.589183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762092.589185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762092.589187:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494011904:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762093.292522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762094.316553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762095.340545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762096.364519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762097.388649:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762097.708606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762097.708617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762097.708622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762097.708633:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2102 00000100:00080000:3.0:1789762097.708666:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762097.708675:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762097.708745:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494012032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762097.708806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762097.708809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762097.708812:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762097.708816:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2102 00000100:00080000:3.0:1789762097.708819:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762097.708822:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762097.708833:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762097.708847:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494012160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762097.708859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494012032:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762097.708864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762097.708869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762097.708871:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762097.708877:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2102 00000100:00080000:3.0:1789762097.708882:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762097.708887:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762097.708893:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762097.708910:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494012288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762097.708923:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762097.708926:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762097.708927:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762097.708931:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2102 00000100:00080000:3.0:1789762097.708934:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762097.708938:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762097.708943:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762097.708949:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762097.708960:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494012416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762097.708990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762097/real 1789762097] req@ffff8eb645148700 x1876699494012032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762097.709008:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494012032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762097.709017:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762097.709021:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762097.709024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494012032:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762097.709048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762097.709064:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762097.709073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762097/real 1789762097] req@ffff8eb64514b480 x1876699494012160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762097.709081:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494012160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762097.709088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762097.709090:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762097.709092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012160:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762097.709099:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012288:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762097.709111:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762097.709117:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762097/real 1789762097] req@ffff8eb645149180 x1876699494012288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762097.709125:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494012288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762097.709131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762097.709133:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762097.709135:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012288:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762097.709138:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494012416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762097.709149:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762097.709155:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762097/real 1789762097] req@ffff8eb64514ad80 x1876699494012416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762097.709163:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494012416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762097.709168:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762097.709170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762097.709172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494012416:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762098.412518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762099.436545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762100.460594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762101.484672:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762102.508583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762102.828542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762102.828556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762102.828561:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762102.828572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2107 00000100:00080000:0.0:1789762102.828596:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762102.828605:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762102.828655:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494012544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762102.828684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762102.828687:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762102.828688:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762102.828692:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2107 00000100:00080000:0.0:1789762102.828695:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762102.828699:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762102.828710:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762102.828720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494012672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762102.828728:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762102.828733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762102.828735:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789762102.828737:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494012544:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762102.828740:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2107 00000100:00080000:0.0:1789762102.828744:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762102.828750:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762102.828753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762102.828761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494012800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762102.828769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762102.828772:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762102.828773:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762102.828776:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2107 00000100:00080000:0.0:1789762102.828778:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762102.828780:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762102.828783:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762102.828790:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494012928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762102.828835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762102.828895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762102/real 1789762102] req@ffff8eb64304e680 x1876699494012544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762102.828912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494012544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762102.828922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762102.828926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762102.828929:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494012544:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762102.828955:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762102.828974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762102.828984:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762102/real 1789762102] req@ffff8eb64304f480 x1876699494012672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762102.828992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494012672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762102.828998:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762102.829001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762102.829003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012672:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762102.829008:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012800:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762102.829020:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762102.829026:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762102/real 1789762102] req@ffff8eb64304d880 x1876699494012800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762102.829034:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494012800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762102.829040:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762102.829041:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762102.829043:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494012800:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762102.829047:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494012928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762102.829058:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762102.829064:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762102/real 1789762102] req@ffff8eb64304d500 x1876699494012928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762102.829071:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494012928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762102.829076:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762102.829078:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762102.829080:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494012928:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762103.532592:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762104.556704:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762105.580579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762106.604663:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762107.628543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762107.948543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762107.948555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762107.948560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762107.948572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2112 00000100:00080000:3.0:1789762107.948585:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762107.948593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762107.948640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494013056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762107.948673:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762107.948677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762107.948680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762107.948685:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2112 00000100:00080000:3.0:1789762107.948690:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762107.948696:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762107.948706:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762107.948721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494013184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762107.948736:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494013056:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762107.948737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762107.948751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762107.948753:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762107.948758:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2112 00000100:00080000:3.0:1789762107.948761:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762107.948765:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762107.948771:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762107.948783:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494013312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762107.948796:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762107.948800:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762107.948802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762107.948807:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2112 00000100:00080000:3.0:1789762107.948811:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762107.948815:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762107.948820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762107.948828:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762107.948840:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494013440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762107.948888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762107/real 1789762107] req@ffff8eb64514ad80 x1876699494013056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762107.948933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494013056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762107.948947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762107.948951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762107.948956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494013056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762107.948986:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762107.949013:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762107.949028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762107/real 1789762107] req@ffff8eb645149180 x1876699494013184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762107.949042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494013184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762107.949065:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762107.949069:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762107.949074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013184:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762107.949081:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013312:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762107.949102:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762107.949153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762107/real 1789762107] req@ffff8eb64514b480 x1876699494013312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762107.949167:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494013312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762107.949176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762107.949179:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762107.949183:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013312:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762107.949189:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494013440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762107.949209:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762107.949220:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762107/real 1789762107] req@ffff8eb645148700 x1876699494013440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762107.949232:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494013440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762107.949241:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762107.949244:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762107.949247:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494013440:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762108.652593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762109.676697:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762110.700548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762111.724593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762112.748573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762113.068512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762113.068523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762113.068528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762113.068538:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2117 00000100:00080000:0.0:1789762113.068555:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762113.068563:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762113.068603:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494013568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762113.068627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762113.068630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762113.068632:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762113.068635:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2117 00000100:00080000:0.0:1789762113.068638:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762113.068662:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762113.068675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762113.068685:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494013568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762113.068692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494013696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762113.068709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762113.068716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762113.068719:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762113.068724:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2117 00000100:00080000:0.0:1789762113.068731:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762113.068737:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762113.068744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762113.068762:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494013824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762113.068789:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762113.068792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762113.068794:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789762113.068799:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762113.068801:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2117 00000100:00080000:0.0:1789762113.068807:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762113.068810:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762113.068815:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762113.068824:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494013952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762113.068879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762113/real 1789762113] req@ffff8eb64304ce00 x1876699494013568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762113.068901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494013568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762113.068913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762113.068917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762113.068922:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494013568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762113.068950:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762113.068974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762113.068986:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762113/real 1789762113] req@ffff8eb64304dc00 x1876699494013696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762113.068997:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494013696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762113.069004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762113.069008:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762113.069011:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013696:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762113.069016:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013824:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762113.069030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762113.069038:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762113/real 1789762113] req@ffff8eb64304c380 x1876699494013824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762113.069048:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494013824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762113.069056:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762113.069058:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762113.069061:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494013824:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762113.069065:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494013952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762113.069109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762113.069139:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762113/real 1789762113] req@ffff8eb64304f800 x1876699494013952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762113.069158:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494013952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762113.069166:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762113.069169:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762113.069172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494013952:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762113.772630:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762114.796542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762115.820579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762116.844547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762117.868561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762118.188509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762118.188525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762118.188529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762118.188538:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2122 00000100:00080000:0.0:1789762118.188548:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762118.188566:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762118.188599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494014080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762118.188618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762118.188631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762118.188633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762118.188637:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2122 00000100:00080000:0.0:1789762118.188642:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762118.188647:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762118.188658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762118.188660:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494014080:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762118.188676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494014208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762118.188702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762118.188705:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762118.188707:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762118.188711:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2122 00000100:00080000:0.0:1789762118.188714:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762118.188717:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762118.188723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789762118.188736:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762118.188741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494014336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762118.188757:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762118.188761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762118.188764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762118.188770:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2122 00000100:00100000:3.0:1789762118.188775:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762118/real 1789762118] req@ffff8eb64304e300 x1876699494014080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762118.188783:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762118.188790:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762118.188794:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494014080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762118.188798:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789762118.188821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762118.188826:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762118.188827:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494014464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762118.188830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494014080:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762118.188860:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762118.188878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762118.188887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762118/real 1789762118] req@ffff8eb64304c000 x1876699494014208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762118.188901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494014208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762118.188917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762118.188919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762118.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014208:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762118.188926:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014336:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762118.188937:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762118.188945:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762118/real 1789762118] req@ffff8eb64304d180 x1876699494014336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762118.188952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494014336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762118.188958:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762118.188960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762118.188962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014336:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762118.188966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494014464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762118.188981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762118.188992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762118/real 1789762118] req@ffff8eb64304ed80 x1876699494014464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762118.189006:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494014464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762118.189023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762118.189027:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762118.189031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494014464:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762118.892574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762119.916529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762120.940574:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762121.964645:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762122.988671:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762123.308521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762123.308532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762123.308537:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762123.308555:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2127 00000100:00080000:3.0:1789762123.308568:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762123.308575:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762123.308625:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494014592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762123.308660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762123.308664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762123.308666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762123.308670:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2127 00000100:00080000:3.0:1789762123.308674:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762123.308679:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762123.308690:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762123.308708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494014720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762123.308710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494014592:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762123.308737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762123.308741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762123.308743:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762123.308747:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2127 00000100:00080000:3.0:1789762123.308751:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762123.308754:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762123.308759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762123.308770:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494014848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762123.308780:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762123.308783:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762123.308785:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762123.308791:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762123.308796:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2127 00000100:00080000:3.0:1789762123.308802:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762123.308806:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762123.308812:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762123.308825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494014976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762123.308868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762123/real 1789762123] req@ffff8eb64514bb80 x1876699494014592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762123.308895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494014592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762123.308909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762123.308914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762123.308919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494014592:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762123.308947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762123.308975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762123.308989:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762123/real 1789762123] req@ffff8eb645148000 x1876699494014720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762123.309051:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494014720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762123.309060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762123.309063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762123.309067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762123.309073:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014848:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762123.309095:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762123.309105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762123/real 1789762123] req@ffff8eb645149880 x1876699494014848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762123.309117:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494014848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762123.309124:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762123.309127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762123.309130:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494014848:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762123.309135:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494014976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762123.309150:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762123.309158:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762123/real 1789762123] req@ffff8eb64514b100 x1876699494014976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762123.309169:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494014976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762123.309176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762123.309179:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762123.309182:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494014976:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762124.012648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762125.036573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762126.060597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762127.084561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762128.108596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762128.428580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762128.428596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762128.428603:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762128.428620:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2132 00000100:00080000:0.0:1789762128.428638:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762128.428676:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762128.428733:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494015104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762128.428796:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762128.428801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762128.428804:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762128.428811:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2132 00000100:00080000:0.0:1789762128.428817:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762128.428823:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762128.428837:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762128.428873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494015104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762128.428876:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494015232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762128.428898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762128.428903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762128.428907:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762128.428912:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2132 00000100:00080000:0.0:1789762128.428918:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762128.428925:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762128.428934:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762128.428950:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494015360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762128.429008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789762128.429009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762128.429017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762128.429021:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762128.429029:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2132 00000100:00080000:0.0:1789762128.429035:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762128.429041:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762128.429066:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762128.429086:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494015488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762128.429089:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762128/real 1789762128] req@ffff8eb64304ed80 x1876699494015104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762128.429131:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494015104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762128.429146:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762128.429150:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762128.429155:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494015104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762128.429185:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762128.429218:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762128.429240:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762128/real 1789762128] req@ffff8eb64304d180 x1876699494015232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762128.429253:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494015232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762128.429261:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762128.429281:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762128.429284:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762128.429289:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015360:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762128.429308:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762128.429317:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762128/real 1789762128] req@ffff8eb64304c000 x1876699494015360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762128.429333:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494015360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762128.429340:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762128.429343:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762128.429345:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015360:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762128.429350:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494015488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762128.429381:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762128.429390:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762128/real 1789762128] req@ffff8eb64304e300 x1876699494015488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762128.429401:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494015488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762128.429407:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762128.429410:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762128.429412:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494015488:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762129.132549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762130.156550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762131.180578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762132.204555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762133.228571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762133.548504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762133.548514:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762133.548519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762133.548530:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2138 00000100:00080000:3.0:1789762133.548542:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762133.548551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762133.548592:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494015616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762133.548624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762133.548627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762133.548628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762133.548632:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2138 00000100:00080000:3.0:1789762133.548634:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762133.548638:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762133.548647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762133.548658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494015744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762133.548667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762133.548669:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494015616:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762133.548671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762133.548675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762133.548684:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2138 00000100:00080000:3.0:1789762133.548689:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762133.548693:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762133.548698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762133.548713:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494015872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762133.548726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762133.548730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762133.548733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762133.548737:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2138 00000100:00080000:3.0:1789762133.548754:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762133.548762:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762133.548763:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762133.548771:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762133.548782:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494016000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762133.548819:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762133/real 1789762133] req@ffff8eb64514b800 x1876699494015616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762133.548840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494015616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762133.548851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762133.548867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762133.548873:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494015616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762133.548903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762133.548943:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762133.548957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762133/real 1789762133] req@ffff8eb64514a300 x1876699494015744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762133.548971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494015744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762133.548980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762133.548983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762133.548988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015744:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762133.548994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015872:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762133.549016:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762133.549028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762133/real 1789762133] req@ffff8eb645148e00 x1876699494015872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762133.549040:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494015872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762133.549049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762133.549052:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762133.549056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494015872:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762133.549062:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494016000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762133.549082:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762133.549094:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762133/real 1789762133] req@ffff8eb645148380 x1876699494016000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762133.549107:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494016000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762133.549115:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762133.549119:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762133.549122:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494016000:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762134.252554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762135.276565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762136.300564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762137.324570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762138.348553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762138.668540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762138.668555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762138.668560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762138.668574:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2143 00000100:00080000:3.0:1789762138.668588:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762138.668598:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762138.668648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494016128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762138.668683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762138.668687:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762138.668689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762138.668693:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2143 00000100:00080000:3.0:1789762138.668697:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762138.668703:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762138.668715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762138.668732:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494016128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762138.668734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494016256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762138.668750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762138.668753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762138.668756:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762138.668760:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2143 00000100:00080000:3.0:1789762138.668764:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762138.668768:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762138.668774:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762138.668786:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494016384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762138.668797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762138.668801:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762138.668802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762138.668807:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2143 00000100:00080000:3.0:1789762138.668811:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762138.668815:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762138.668820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762138.668832:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494016512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762138.668868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762138.668925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016384:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762138.668953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494016512:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762139.372532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762140.396562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762141.420573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762141.740609:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762141.740645:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762141.740659:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762141.740687:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762141.740694:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762141.740707:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762141.740726:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762138/real 1789762141] req@ffff8eb645149500 x1876699494016256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.740755:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494016256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.740768:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.740772:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762141.740776:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762141.740787:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762138/real 1789762141] req@ffff8eb645148a80 x1876699494016384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.740799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494016384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.740806:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.740810:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762141.740813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016384:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762141.740820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762138/real 1789762141] req@ffff8eb64514a680 x1876699494016512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.740833:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494016512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.740841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.740845:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762141.740848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494016512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762141.740855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762138/real 1789762141] req@ffff8eb645149c00 x1876699494016128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.740897:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494016128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.740906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.740910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762141.740913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494016128:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789762142.444597:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762143.468586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762143.788471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762143.788481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762143.788485:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762143.788495:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2148 00000100:00080000:3.0:1789762143.788512:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762143.788520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762143.788558:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494016640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762143.788585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762143.788588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762143.788590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762143.788595:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2148 00000100:00080000:3.0:1789762143.788598:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762143.788601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762143.788611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762143.788621:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494016768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762143.788632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762143.788636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762143.788638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762143.788642:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2148 00000100:00080000:3.0:1789762143.788646:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762143.788650:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762143.788655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762143.788664:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494016640:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762143.788668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494016896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762143.788681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762143.788685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762143.788687:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762143.788691:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2148 00000100:00080000:3.0:1789762143.788694:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762143.788698:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762143.788702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762143.788713:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494017024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762143.788735:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762143.788778:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762143/real 1789762143] req@ffff8eb645149c00 x1876699494016640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762143.788796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494016640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762143.788806:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762143.788809:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762143.788813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494016640:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762143.788821:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762143.788861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762143.788875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762143/real 1789762143] req@ffff8eb64514a680 x1876699494016768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762143.788887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494016768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762143.788896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762143.788900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762143.788904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016768:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762143.788929:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016896:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762143.788952:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762143.788965:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762143/real 1789762143] req@ffff8eb645148a80 x1876699494016896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762143.788979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494016896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762143.788987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762143.788991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762143.789006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494016896:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762143.789012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494017024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762143.789037:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762143.789049:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762143/real 1789762143] req@ffff8eb645149500 x1876699494017024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762143.789060:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494017024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762143.789067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762143.789070:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762143.789073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494017024:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762144.492599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762145.516626:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762146.540545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762147.564558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762148.588589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762148.908478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762148.908492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762148.908497:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762148.908512:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2153 00000100:00080000:3.0:1789762148.908526:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762148.908534:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762148.908576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494017152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762148.908603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762148.908606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762148.908609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762148.908612:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2153 00000100:00080000:3.0:1789762148.908615:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762148.908618:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762148.908627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762148.908638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494017280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762148.908646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762148.908649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762148.908650:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762148.908652:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2153 00000100:00080000:3.0:1789762148.908654:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762148.908658:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762148.908661:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762148.908670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494017408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762148.908682:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494017152:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762148.908683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762148.908687:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762148.908692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762148.908703:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2153 00000100:00080000:3.0:1789762148.908705:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762148.908708:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762148.908711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762148.908724:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494017536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762148.908776:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762148.908832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762148/real 1789762148] req@ffff8eb645148380 x1876699494017152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762148.908859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494017152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762148.908872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762148.908876:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762148.908881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494017152:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762148.908889:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762148.908912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762148.908924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762148/real 1789762148] req@ffff8eb645148e00 x1876699494017280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762148.908937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494017280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762148.908945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762148.908949:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762148.908953:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017280:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762148.908979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017408:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762148.908998:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762148.909008:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762148/real 1789762148] req@ffff8eb64514a300 x1876699494017408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762148.909020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494017408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762148.909028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762148.909032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762148.909036:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017408:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762148.909042:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494017536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762148.909059:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762148.909069:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762148/real 1789762148] req@ffff8eb64514b800 x1876699494017536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762148.909080:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494017536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762148.909088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762148.909091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762148.909095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494017536:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762149.612542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762150.636574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762151.660544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762152.684584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762153.708526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762154.028538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762154.028552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762154.028557:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762154.028570:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2158 00000100:00080000:3.0:1789762154.028583:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762154.028591:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762154.028636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494017664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762154.028665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762154.028668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762154.028670:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762154.028673:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2158 00000100:00080000:3.0:1789762154.028676:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762154.028678:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762154.028688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762154.028698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494017792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762154.028706:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762154.028709:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762154.028713:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762154.028715:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494017664:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762154.028718:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2158 00000100:00080000:3.0:1789762154.028725:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762154.028728:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762154.028731:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762154.028739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494017920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762154.028747:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762154.028750:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762154.028751:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762154.028754:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2158 00000100:00080000:3.0:1789762154.028756:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762154.028758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762154.028761:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762154.028789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494018048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762154.028809:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762154.028855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762154/real 1789762154] req@ffff8eb64514b100 x1876699494017664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762154.028870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494017664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762154.028879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762154.028883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762154.028886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494017664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762154.028910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762154.028929:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762154.028937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762154/real 1789762154] req@ffff8eb645149880 x1876699494017792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762154.028946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494017792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762154.028952:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762154.028954:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762154.028956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017792:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762154.028960:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017920:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762154.028972:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762154.028977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762154/real 1789762154] req@ffff8eb645148000 x1876699494017920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762154.028985:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494017920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762154.028991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762154.028992:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762154.028994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494017920:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762154.028998:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494018048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762154.029008:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762154.029015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762154/real 1789762154] req@ffff8eb64514bb80 x1876699494018048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762154.029027:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494018048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762154.029033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762154.029034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762154.029036:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494018048:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762154.732605:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762155.756563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762156.780621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762157.804550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762158.828589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762159.148489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762159.148500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762159.148505:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762159.148513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2163 00000100:00080000:0.0:1789762159.148523:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762159.148529:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762159.148559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494018176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762159.148582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762159.148585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762159.148586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762159.148589:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2163 00000100:00080000:0.0:1789762159.148592:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762159.148595:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762159.148603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762159.148615:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494018176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762159.148616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494018304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762159.148631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762159.148633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762159.148634:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762159.148637:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2163 00000100:00080000:0.0:1789762159.148640:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762159.148641:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762159.148645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762159.148654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494018432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762159.148661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762159.148664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762159.148665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762159.148668:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2163 00000100:00080000:0.0:1789762159.148670:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762159.148675:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762159.148676:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762159.148683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762159.148690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494018560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762159.148717:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762159/real 1789762159] req@ffff8eb64304f800 x1876699494018176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762159.148760:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494018176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762159.148771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762159.148776:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762159.148780:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494018176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762159.148802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762159.148821:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762159.148831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762159/real 1789762159] req@ffff8eb64304c380 x1876699494018304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762159.148839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494018304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762159.148844:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762159.148847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762159.148849:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018304:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762159.148852:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018432:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762159.148863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762159.148870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762159/real 1789762159] req@ffff8eb64304dc00 x1876699494018432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762159.148878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494018432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762159.148887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762159.148889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762159.148891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018432:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762159.148894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494018560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762159.148905:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762159.148911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762159/real 1789762159] req@ffff8eb64304ce00 x1876699494018560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762159.148918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494018560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762159.148923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762159.148925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762159.148927:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494018560:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762159.852597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762160.876536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762161.900593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762162.924561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762163.948551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762164.268542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762164.268553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762164.268569:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762164.268582:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2168 00000100:00080000:1.0:1789762164.268594:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762164.268602:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762164.268651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494018688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762164.268676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762164.268679:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762164.268680:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762164.268684:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2168 00000100:00080000:1.0:1789762164.268686:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762164.268709:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762164.268732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762164.268744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494018688:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762164.268745:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494018816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762164.268759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762164.268762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762164.268764:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762164.268766:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2168 00000100:00080000:1.0:1789762164.268770:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762164.268772:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762164.268777:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762164.268786:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494018944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762164.268798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762164.268801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762164.268803:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762164.268805:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2168 00000100:00080000:1.0:1789762164.268807:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762164.268810:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762164.268813:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762164.268820:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494019072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762164.268824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762164.268870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff8eb65016d500 x1876699494018688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762164.268885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494018688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762164.268900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762164.268903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762164.268906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494018688:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762164.268929:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762164.268944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762164.268953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff8eb65016e300 x1876699494018816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762164.268961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494018816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762164.268967:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762164.268969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762164.268971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018816:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762164.268974:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018944:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762164.268986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762164.268992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff8eb65016f800 x1876699494018944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762164.269000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494018944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762164.269005:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762164.269007:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762164.269009:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494018944:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762164.269013:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494019072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762164.269024:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762164.269030:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762164/real 1789762164] req@ffff8eb65016e680 x1876699494019072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762164.269038:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494019072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762164.269043:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762164.269044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762164.269046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494019072:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762164.972634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762165.996592:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762167.020573:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762168.044578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762169.068610:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762169.388609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762169.388642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762169.388666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762169.388681:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2173 00000100:00080000:3.0:1789762169.388710:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762169.388716:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762169.388746:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494019200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762169.388770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762169.388773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762169.388775:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762169.388778:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2173 00000100:00080000:3.0:1789762169.388781:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762169.388783:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762169.388791:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762169.388800:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494019328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762169.388808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762169.388811:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762169.388812:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762169.388814:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2173 00000100:00080000:3.0:1789762169.388816:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762169.388820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494019200:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762169.388821:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762169.388829:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762169.388836:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494019456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762169.388844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762169.388846:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762169.388848:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762169.388851:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2173 00000100:00080000:3.0:1789762169.388854:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762169.388856:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762169.388860:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762169.388866:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494019584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762169.388882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762169.388916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff8eb645148700 x1876699494019200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762169.388929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494019200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762169.388937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762169.388939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762169.388942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494019200:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762169.388947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762169.388964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762169.388976:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff8eb64514b480 x1876699494019328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762169.388989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494019328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762169.388998:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762169.389001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762169.389005:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019328:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762169.389029:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019456:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762169.389046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762169.389053:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff8eb645149180 x1876699494019456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762169.389066:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494019456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762169.389071:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762169.389073:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762169.389075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019456:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762169.389078:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494019584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762169.389088:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762169.389094:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762169/real 1789762169] req@ffff8eb64514ad80 x1876699494019584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762169.389101:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494019584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762169.389106:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762169.389108:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762169.389110:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494019584:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762170.092633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762171.116554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762172.140657:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762173.164554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762174.188570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762174.508549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762174.508560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762174.508564:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762174.508575:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2179 00000100:00080000:1.0:1789762174.508587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762174.508594:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762174.508667:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494019712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762174.508701:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762174.508705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762174.508708:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762174.508717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2179 00000100:00080000:1.0:1789762174.508722:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762174.508726:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762174.508738:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762174.508752:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494019840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762174.508764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762174.508768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762174.508770:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762174.508774:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2179 00000100:00080000:1.0:1789762174.508777:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762174.508796:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762174.508816:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494019712:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762174.508822:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762174.508839:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494019968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762174.508852:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762174.508856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762174.508858:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762174.508863:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2179 00000100:00080000:1.0:1789762174.508868:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762174.508873:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762174.508878:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762174.508890:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494020096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762174.508917:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762174.508999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff8eb65016d880 x1876699494019712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.509045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494019712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.509059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.509063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762174.509068:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494019712:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762174.509107:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762174.509147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762174.509169:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff8eb65016ea00 x1876699494019840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.509191:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494019840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.509201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.509204:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762174.509208:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019840:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762174.509247:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019968:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762174.509274:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762174.509286:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff8eb65016e680 x1876699494019968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.509298:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494019968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.509307:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.509310:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762174.509313:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494019968:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762174.509320:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494020096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762174.509354:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762174.509365:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762174/real 1789762174] req@ffff8eb65016f800 x1876699494020096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762174.509378:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494020096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762174.509386:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762174.509389:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762174.509392:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494020096:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762175.212627:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762176.236695:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762177.260602:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762178.284566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762179.308691:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762179.628528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762179.628537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762179.628542:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762179.628566:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2184 00000100:00080000:1.0:1789762179.628588:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762179.628598:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762179.628637:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494020224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762179.628665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762179.628668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762179.628669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762179.628673:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2184 00000100:00080000:1.0:1789762179.628676:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762179.628680:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762179.628687:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762179.628705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494020352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762179.628716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762179.628720:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494020224:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762179.628720:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762179.628723:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762179.628731:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2184 00000100:00080000:1.0:1789762179.628734:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762179.628736:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762179.628740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762179.628749:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494020480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762179.628762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762179.628764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762179.628766:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762179.628768:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2184 00000100:00080000:1.0:1789762179.628771:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762179.628773:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762179.628776:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762179.628782:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762179.628786:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494020608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762179.628820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff8eb65016e300 x1876699494020224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762179.628833:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494020224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762179.628840:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762179.628843:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762179.628846:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494020224:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762179.628863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762179.628880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762179.628892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff8eb65016d500 x1876699494020352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762179.628905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494020352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762179.628913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762179.628917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762179.628920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762179.628927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020480:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762179.628947:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762179.628956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff8eb65016fb80 x1876699494020480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762179.628964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494020480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762179.628969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762179.628971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762179.628974:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020480:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762179.628977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494020608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762179.628989:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762179.628995:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762179/real 1789762179] req@ffff8eb65016c380 x1876699494020608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762179.629006:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494020608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762179.629011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762179.629013:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762179.629015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494020608:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762180.332607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762181.356611:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762182.380635:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762183.404526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762184.428609:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762184.748508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.748524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762184.748529:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.748555:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2189 00000100:00080000:0.0:1789762184.748570:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.748580:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762184.748630:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494020736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.748667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.748671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.748674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.748679:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2189 00000100:00080000:0.0:1789762184.748683:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.748689:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762184.748702:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762184.748719:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494020736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762184.748721:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494020864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.748739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.748743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.748745:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.748761:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2189 00000100:00080000:0.0:1789762184.748771:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.748776:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762184.748782:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789762184.748793:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762184.748799:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494020992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762184.748813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762184.748817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762184.748820:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762184.748825:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2189 00000100:00080000:0.0:1789762184.748829:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762184.748834:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762184.748841:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762184.748855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff8eb64304d500 x1876699494020736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762184.748858:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494021120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762184.748902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494020736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762184.748913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762184.748916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762184.748920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494020736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762184.748945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762184.748964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762184.748974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff8eb64304d880 x1876699494020864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762184.748983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494020864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762184.748990:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762184.748992:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762184.748995:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020864:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762184.748998:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020992:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762184.749011:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762184.749024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff8eb64304f480 x1876699494020992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762184.749033:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494020992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762184.749038:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762184.749041:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762184.749045:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494020992:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762184.749051:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494021120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762184.749072:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762184.749084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762184/real 1789762184] req@ffff8eb64304e680 x1876699494021120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762184.749098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494021120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762184.749112:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762184.749114:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762184.749117:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494021120:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762185.452580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762186.476601:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762187.500582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762188.524561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762189.548525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762189.868480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762189.868494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762189.868511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762189.868525:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2194 00000100:00080000:0.0:1789762189.868538:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762189.868547:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762189.868594:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494021248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762189.868629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762189.868633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762189.868635:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762189.868640:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2194 00000100:00080000:0.0:1789762189.868643:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762189.868648:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762189.868660:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762189.868674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494021376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762189.868677:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494021248:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762189.868694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762189.868698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762189.868700:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762189.868706:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2194 00000100:00080000:0.0:1789762189.868711:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762189.868715:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762189.868721:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762189.868734:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494021504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762189.868751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762189.868756:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789762189.868757:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762189.868760:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762189.868767:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2194 00000100:00080000:0.0:1789762189.868771:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762189.868776:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762189.868782:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762189.868796:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494021632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762189.868806:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762189/real 1789762189] req@ffff8eb64304ca80 x1876699494021248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762189.868823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494021248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762189.868832:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762189.868835:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762189.868839:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494021248:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762189.868861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762189.868877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762189.868885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762189/real 1789762189] req@ffff8eb64304ea00 x1876699494021376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762189.868893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494021376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762189.868898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762189.868901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762189.868903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021376:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762189.868908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021504:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762189.868919:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762189.868925:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762189/real 1789762189] req@ffff8eb64304c700 x1876699494021504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762189.868933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494021504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762189.868938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762189.868940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762189.868942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021504:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762189.868945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494021632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762189.868956:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762189.868961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762189/real 1789762189] req@ffff8eb64304f100 x1876699494021632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762189.868969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494021632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762189.868974:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762189.868976:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762189.868978:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494021632:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762190.572547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762191.596613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762192.620566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762193.644579:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762194.668536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762194.988556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762194.988572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762194.988577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762194.988591:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2199 00000100:00080000:3.0:1789762194.988606:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762194.988615:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762194.988663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494021760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762194.988703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762194.988707:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762194.988710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762194.988715:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2199 00000100:00080000:3.0:1789762194.988720:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762194.988724:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762194.988736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762194.988748:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494021888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762194.988757:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762194.988763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762194.988766:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762194.988768:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494021760:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762194.988772:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2199 00000100:00080000:3.0:1789762194.988781:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762194.988788:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762194.988794:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762194.988811:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494022016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762194.988823:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762194.988827:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762194.988829:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762194.988835:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2199 00000100:00080000:3.0:1789762194.988840:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762194.988845:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762194.988850:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762194.988862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762194.988867:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494022144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762194.988924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762194/real 1789762194] req@ffff8eb64514ad80 x1876699494021760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762194.988947:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494021760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762194.988960:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762194.988965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762194.988970:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494021760:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762194.988996:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762194.989022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762194.989036:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762194/real 1789762194] req@ffff8eb645149180 x1876699494021888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762194.989050:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494021888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762194.989060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762194.989064:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762194.989068:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494021888:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762194.989077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022016:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762194.989099:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762194.989110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762194/real 1789762194] req@ffff8eb64514b480 x1876699494022016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762194.989123:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494022016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762194.989132:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762194.989136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762194.989140:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022016:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762194.989147:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494022144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762194.989167:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762194.989179:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762194/real 1789762194] req@ffff8eb645148700 x1876699494022144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762194.989202:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494022144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762194.989214:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762194.989218:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762194.989222:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494022144:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762195.692571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762196.716560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762197.740596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762198.764593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762199.788609:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762200.108505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762200.108518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762200.108522:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762200.108533:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2204 00000100:00080000:3.0:1789762200.108545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762200.108554:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762200.108599:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494022272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762200.108627:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762200.108631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762200.108632:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762200.108637:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2204 00000100:00080000:3.0:1789762200.108639:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762200.108642:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762200.108653:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762200.108663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494022400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762200.108673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762200.108676:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762200.108677:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494022272:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762200.108678:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762200.108686:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2204 00000100:00080000:3.0:1789762200.108690:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762200.108692:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762200.108695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762200.108705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494022528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762200.108719:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762200.108722:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762200.108724:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762200.108727:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2204 00000100:00080000:3.0:1789762200.108730:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762200.108733:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762200.108736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762200.108744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494022656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762200.108756:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762200.108802:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff8eb64514bb80 x1876699494022272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.108823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494022272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.108836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.108841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762200.108845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494022272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762200.108874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.108894:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762200.108905:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff8eb645148000 x1876699494022400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.108913:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494022400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.108920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.108923:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762200.108925:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022400:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762200.108930:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022528:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762200.108943:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762200.108949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff8eb645149880 x1876699494022528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.108957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494022528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.108963:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.108964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762200.108966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022528:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762200.108970:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494022656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762200.108981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762200.108987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762200/real 1789762200] req@ffff8eb64514b100 x1876699494022656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762200.108995:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494022656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762200.109000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762200.109001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762200.109011:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494022656:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762200.812535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762201.836555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762202.860538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762203.884523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762204.908541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762205.228514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762205.228525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762205.228530:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762205.228540:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2209 00000100:00080000:0.0:1789762205.228552:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762205.228560:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762205.228605:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494022784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762205.228630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762205.228633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762205.228635:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762205.228638:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2209 00000100:00080000:0.0:1789762205.228641:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762205.228644:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762205.228654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762205.228667:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494022912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762205.228667:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494022784:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762205.228681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762205.228684:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762205.228685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762205.228687:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2209 00000100:00080000:0.0:1789762205.228689:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762205.228691:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762205.228694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762205.228703:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494023040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762205.228710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762205.228713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762205.228714:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762205.228717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2209 00000100:00080000:0.0:1789762205.228720:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762205.228722:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762205.228725:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762205.228731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494023168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762205.228787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762205.228812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023040:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762205.228832:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494023168:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762205.932566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762206.956544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762207.980579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762208.300753:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762208.300812:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762208.300836:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762208.300893:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762208.300903:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762208.300910:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:3.0:1789762208.300988:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762208] req@ffff8eb64304c700 x1876699494022912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762208.301026:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494022912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762208.301042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762208.301049:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762208.301056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494022912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762208.301066:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762208] req@ffff8eb64304ea00 x1876699494023040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762208.301082:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494023040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762208.301099:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762208.301103:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762208.301107:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023040:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762208.301115:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762208] req@ffff8eb64304ca80 x1876699494023168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762208.301128:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494023168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762208.301137:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762208.301141:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762208.301145:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494023168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762208.301153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762205/real 1789762208] req@ffff8eb64304f100 x1876699494022784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762208.301168:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494022784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762208.301176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762208.301180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762208.301184:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494022784:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762209.004574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762210.028548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762210.348555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.348570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762210.348576:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.348590:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2214 00000100:00080000:3.0:1789762210.348604:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.348615:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762210.348663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494023296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.348704:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.348708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762210.348711:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.348721:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2214 00000100:00080000:3.0:1789762210.348726:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.348732:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762210.348745:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762210.348760:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494023296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762210.348765:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494023424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.348781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.348785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762210.348787:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.348792:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2214 00000100:00080000:3.0:1789762210.348796:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.348801:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762210.348807:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762210.348821:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494023552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762210.348841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762210.348845:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762210.348848:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762210.348853:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2214 00000800:00000100:0.0:1789762210.348858:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762210.348861:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762210.348869:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762210.348875:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762210.348890:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494023680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762210.348920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff8eb64514b800 x1876699494023296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762210.348943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494023296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762210.348956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762210.348961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762210.348965:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494023296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762210.348999:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762210.349027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762210.349048:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff8eb64514a300 x1876699494023424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762210.349061:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494023424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762210.349069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762210.349073:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762210.349076:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023424:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762210.349083:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023552:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762210.349105:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762210.349117:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff8eb645148e00 x1876699494023552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762210.349129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494023552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762210.349138:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762210.349141:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762210.349145:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023552:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762210.349150:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494023680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762210.349172:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762210.349184:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762210/real 1789762210] req@ffff8eb645148380 x1876699494023680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762210.349210:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494023680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762210.349219:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762210.349223:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762210.349226:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494023680:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762211.052639:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762212.076552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762213.100578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762214.124551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762215.148593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762215.468541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762215.468555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762215.468561:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762215.468574:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2219 00000100:00080000:0.0:1789762215.468589:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762215.468599:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762215.468648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494023808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762215.468681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762215.468685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762215.468688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762215.468692:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2219 00000100:00080000:0.0:1789762215.468697:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762215.468703:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762215.468716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762215.468734:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494023936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762215.468737:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494023808:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762215.468755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762215.468759:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762215.468762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762215.468766:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2219 00000100:00080000:0.0:1789762215.468770:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762215.468775:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762215.468781:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762215.468796:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494024064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762215.468807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762215.468812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762215.468814:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762215.468818:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2219 00000100:00080000:0.0:1789762215.468822:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762215.468829:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:3.0:1789762215.468830:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789762215.468837:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762215.468852:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494024192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762215.468887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff8eb64304e680 x1876699494023808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762215.468917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494023808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.468931:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762215.468936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762215.468941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494023808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762215.468986:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762215.469015:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762215.469029:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff8eb64304f480 x1876699494023936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762215.469043:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494023936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.469052:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762215.469056:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762215.469060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494023936:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762215.469067:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024064:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762215.469091:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762215.469104:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff8eb64304d880 x1876699494024064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762215.469116:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494024064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.469125:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762215.469129:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762215.469133:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024064:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762215.469139:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494024192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762215.469161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762215.469174:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762215/real 1789762215] req@ffff8eb64304d500 x1876699494024192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762215.469194:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494024192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762215.469203:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762215.469207:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762215.469210:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494024192:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762216.172564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762217.196540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762218.220571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762219.244626:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762220.268585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762220.588529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.588547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762220.588551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.588563:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2225 00000100:00080000:3.0:1789762220.588577:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.588586:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762220.588643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494024320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762220.588669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.588672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.588674:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.588678:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2225 00000100:00080000:3.0:1789762220.588681:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.588684:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762220.588694:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.588706:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494024448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762220.588713:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494024320:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762220.588716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.588724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.588728:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.588732:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2225 00000100:00080000:3.0:1789762220.588734:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.588736:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762220.588745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762220.588754:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494024576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762220.588762:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762220.588764:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762220.588769:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762220.588773:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2225 00000100:00080000:3.0:1789762220.588776:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762220.588778:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762220.588781:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762220.588789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494024704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762220.588804:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762220.588864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff8eb645149500 x1876699494024320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762220.588885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494024320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762220.588898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762220.588902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762220.588907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494024320:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762220.588934:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762220.588957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762220.588969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff8eb645148a80 x1876699494024448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762220.588983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494024448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762220.588992:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762220.588996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762220.588999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024448:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762220.589006:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024576:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762220.589024:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762220.589034:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff8eb64514a680 x1876699494024576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762220.589054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494024576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762220.589063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762220.589066:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762220.589069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024576:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762220.589075:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494024704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762220.589093:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762220.589103:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762220/real 1789762220] req@ffff8eb645149c00 x1876699494024704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762220.589115:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494024704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762220.589124:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762220.589128:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762220.589132:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494024704:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762221.292547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762222.316588:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762223.340578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762224.364536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762225.388580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762225.708491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762225.708499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762225.708503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762225.708512:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2230 00000100:00080000:3.0:1789762225.708523:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762225.708530:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762225.708568:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494024832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762225.708592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762225.708595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762225.708596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762225.708599:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2230 00000100:00080000:3.0:1789762225.708602:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762225.708605:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762225.708612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762225.708623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494024960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762225.708634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762225.708638:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762225.708640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762225.708645:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2230 00000100:00080000:3.0:1789762225.708649:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762225.708654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762225.708661:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762225.708670:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494024832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762225.708676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494025088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762225.708691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762225.708695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762225.708698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762225.708702:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2230 00000100:00080000:3.0:1789762225.708708:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762225.708713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762225.708719:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762225.708730:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762225.708736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494025216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762225.708767:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762225] req@ffff8eb645149c00 x1876699494024832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762225.708786:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494024832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762225.708794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762225.708796:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762225.708799:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494024832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762225.708804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762225.708818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762225.708826:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762225] req@ffff8eb64514a680 x1876699494024960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762225.708834:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494024960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762225.708839:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762225.708841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762225.708843:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494024960:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762225.708861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025088:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762225.708872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762225.708877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762225] req@ffff8eb645148a80 x1876699494025088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762225.708884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494025088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762225.708904:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762225.708906:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762225.708908:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025088:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762225.708912:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494025216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762225.708931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762225.708942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762225/real 1789762225] req@ffff8eb645149500 x1876699494025216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762225.708956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494025216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762225.708965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762225.708968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762225.708972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494025216:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762226.412568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762227.436545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762228.460497:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762229.484567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762230.508591:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762230.828544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762230.828555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762230.828560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762230.828571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2235 00000100:00080000:3.0:1789762230.828583:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762230.828592:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762230.828627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494025344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762230.828650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762230.828652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762230.828654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762230.828658:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2235 00000100:00080000:3.0:1789762230.828661:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762230.828665:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762230.828680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762230.828697:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494025344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762230.828698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494025472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762230.828725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762230.828728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762230.828729:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762230.828734:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2235 00000100:00080000:3.0:1789762230.828737:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762230.828740:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762230.828745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762230.828761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494025600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762230.828772:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762230.828777:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762230.828779:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762230.828783:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2235 00000100:00080000:3.0:1789762230.828788:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762230.828793:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762230.828801:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762230.828810:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762230.828825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494025728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762230.828845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff8eb645148380 x1876699494025344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762230.828863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494025344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762230.828877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762230.828880:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762230.828884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494025344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762230.828907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762230.828924:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762230.828932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff8eb645148e00 x1876699494025472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762230.828941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494025472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762230.828947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762230.828949:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762230.828951:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025472:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762230.828956:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025600:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762230.828967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762230.828973:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff8eb64514a300 x1876699494025600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762230.828980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494025600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762230.828986:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762230.828988:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762230.828990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025600:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762230.828994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494025728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762230.829005:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762230.829012:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762230/real 1789762230] req@ffff8eb64514b800 x1876699494025728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762230.829019:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494025728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762230.829024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762230.829026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762230.829028:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494025728:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762231.532546:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762232.556603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762233.580556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762234.604560:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762235.628586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762235.948576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762235.948589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762235.948595:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762235.948608:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2240 00000100:00080000:2.0:1789762235.948639:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762235.948648:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762235.948716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494025856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762235.948755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762235.948759:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762235.948762:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762235.948767:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2240 00000100:00080000:2.0:1789762235.948772:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762235.948796:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762235.948812:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762235.948833:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494025984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762235.948848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494025856:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762235.948871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762235.948882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762235.948885:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762235.948891:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2240 00000100:00080000:2.0:1789762235.948895:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762235.948899:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762235.948906:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762235.948937:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494026112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762235.948951:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762235.948956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762235.948959:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762235.948963:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2240 00000100:00080000:2.0:1789762235.948973:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762235.948979:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762235.949002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789762235.949003:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762235.949016:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494026240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762235.949052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762235/real 1789762235] req@ffff8eb650f83800 x1876699494025856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762235.949068:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494025856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762235.949077:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762235.949080:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762235.949083:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494025856:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762235.949106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762235.949123:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762235.949131:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762235/real 1789762235] req@ffff8eb650f81180 x1876699494025984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762235.949140:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494025984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762235.949149:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762235.949167:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762235.949171:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494025984:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762235.949178:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026112:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762235.949200:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762235.949212:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762235/real 1789762235] req@ffff8eb650f83b80 x1876699494026112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762235.949223:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494026112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762235.949229:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762235.949232:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762235.949234:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026112:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762235.949238:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494026240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762235.949253:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762235.949260:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762235/real 1789762235] req@ffff8eb650f81f80 x1876699494026240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762235.949273:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494026240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762235.949278:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762235.949280:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762235.949282:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494026240:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762236.652632:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762237.676640:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762238.700557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762239.724621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762240.748700:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762241.068556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.068564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762241.068568:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.068575:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2245 00000100:00080000:3.0:1789762241.068585:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.068591:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762241.068639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494026368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.068677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.068680:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.068681:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.068685:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2245 00000100:00080000:3.0:1789762241.068687:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.068690:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762241.068698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762241.068707:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494026496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.068715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.068718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.068719:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.068721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2245 00000100:00080000:3.0:1789762241.068723:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.068727:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762241.068731:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762241.068737:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494026368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762241.068741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494026624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762241.068753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762241.068756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762241.068757:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762241.068759:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2245 00000100:00080000:3.0:1789762241.068761:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762241.068764:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762241.068767:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762241.068774:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494026752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762241.068828:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762241.068879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff8eb64514b100 x1876699494026368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762241.068894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494026368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762241.068902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762241.068905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762241.068909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494026368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762241.068914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762241.068932:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762241.068956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff8eb645149880 x1876699494026496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762241.068969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494026496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762241.068975:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762241.068977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762241.068979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026496:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762241.068997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026624:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762241.069027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762241.069033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff8eb645148000 x1876699494026624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762241.069041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494026624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762241.069047:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762241.069049:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762241.069051:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494026624:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762241.069055:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494026752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762241.069065:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762241.069072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762241/real 1789762241] req@ffff8eb64514bb80 x1876699494026752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762241.069079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494026752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762241.069089:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762241.069091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762241.069093:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494026752:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762241.772614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762242.796563:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762243.820616:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762244.844716:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762245.868497:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762246.188485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.188493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762246.188497:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.188506:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2250 00000100:00080000:0.0:1789762246.188516:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.188522:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762246.188552:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494026880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.188573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.188575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.188577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.188580:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2250 00000100:00080000:0.0:1789762246.188583:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.188588:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762246.188597:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762246.188602:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494026880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762246.188614:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494027008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.188629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.188633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.188636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.188639:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2250 00000100:00080000:0.0:1789762246.188643:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.188645:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762246.188650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789762246.188659:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762246.188662:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494027136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762246.188671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762246.188674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762246.188676:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762246.188679:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2250 00000100:00080000:0.0:1789762246.188681:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762246.188683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762246.188686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762246.188699:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff8eb64304ce00 x1876699494026880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762246.188706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494027264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762246.188718:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494026880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.188727:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.188730:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762246.188734:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494026880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762246.188756:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.188771:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762246.188778:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff8eb64304dc00 x1876699494027008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.188786:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494027008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.188792:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.188794:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762246.188797:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762246.188800:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027136:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762246.188810:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762246.188816:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff8eb64304c380 x1876699494027136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.188823:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494027136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.188829:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.188831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762246.188832:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027136:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762246.188836:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494027264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762246.188845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762246.188851:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762246/real 1789762246] req@ffff8eb64304f800 x1876699494027264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762246.188859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494027264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762246.188863:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762246.188865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762246.188867:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494027264:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762246.892519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762247.916559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762248.940536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762249.964554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762250.988519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762251.308495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762251.308503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762251.308507:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762251.308515:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2255 00000100:00080000:0.0:1789762251.308524:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762251.308530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762251.308572:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494027392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762251.308594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762251.308596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762251.308598:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762251.308601:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2255 00000100:00080000:0.0:1789762251.308603:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762251.308605:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762251.308613:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762251.308621:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494027520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762251.308630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762251.308635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789762251.308635:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494027392:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762251.308640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762251.308647:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2255 00000100:00080000:0.0:1789762251.308651:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762251.308654:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762251.308661:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762251.308674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494027648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762251.308692:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762251.308695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762251.308696:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762251.308700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2255 00000100:00080000:0.0:1789762251.308703:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762251.308706:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:3.0:1789762251.308710:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789762251.308712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762251.308724:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494027776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762251.308747:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff8eb64304e300 x1876699494027392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762251.308761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494027392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762251.308769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762251.308771:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762251.308774:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494027392:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762251.308793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762251.308809:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762251.308816:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff8eb64304c000 x1876699494027520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762251.308824:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494027520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762251.308830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762251.308832:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762251.308834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027520:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762251.308838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027648:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762251.308848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762251.308854:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff8eb64304d180 x1876699494027648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762251.308861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494027648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762251.308866:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762251.308868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762251.308869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494027648:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762251.308873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494027776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762251.308882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762251.308888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762251/real 1789762251] req@ffff8eb64304ed80 x1876699494027776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762251.308895:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494027776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762251.308900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762251.308901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762251.308904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494027776:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762252.012558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762253.036567:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762254.060570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762255.084567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762256.108603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762256.428507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762256.428517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762256.428521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762256.428531:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2260 00000100:00080000:3.0:1789762256.428541:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762256.428547:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762256.428591:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494027904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762256.428617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762256.428620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762256.428622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762256.428626:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2260 00000100:00080000:3.0:1789762256.428630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762256.428634:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762256.428643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762256.428659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494028032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762256.428660:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494027904:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762256.428678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762256.428682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762256.428684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762256.428688:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2260 00000100:00080000:3.0:1789762256.428692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762256.428695:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762256.428700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762256.428712:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494028160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762256.428721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762256.428725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762256.428727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762256.428736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2260 00000800:00000100:0.0:1789762256.428737:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762256.428744:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762256.428752:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762256.428759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762256.428777:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494028288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762256.428792:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff8eb645148700 x1876699494027904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762256.428817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494027904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762256.428827:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762256.428831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762256.428835:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494027904:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762256.428857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762256.428879:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762256.428890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff8eb64514b480 x1876699494028032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762256.428900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494028032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762256.428907:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762256.428910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762256.428913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028032:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762256.428917:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028160:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762256.428930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762256.428937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff8eb645149180 x1876699494028160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762256.428946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494028160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762256.428953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762256.428955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762256.428957:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028160:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762256.428961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494028288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762256.428974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762256.428981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762256/real 1789762256] req@ffff8eb64514ad80 x1876699494028288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762256.429028:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494028288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762256.429054:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762256.429056:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762256.429060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494028288:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762257.132543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762258.156571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762259.180540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762260.204603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762261.228561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762261.548563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762261.548575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762261.548580:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762261.548594:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2266 00000100:00080000:0.0:1789762261.548606:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762261.548615:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762261.548661:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494028416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762261.548689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762261.548692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762261.548694:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762261.548697:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2266 00000100:00080000:0.0:1789762261.548700:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762261.548702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762261.548713:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762261.548723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494028544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762261.548732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762261.548735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762261.548736:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762261.548739:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2266 00000100:00080000:0.0:1789762261.548741:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762261.548746:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762261.548748:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494028416:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789762261.548750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762261.548762:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494028672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762261.548770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762261.548773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762261.548774:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762261.548777:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2266 00000100:00080000:0.0:1789762261.548779:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762261.548782:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762261.548785:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762261.548792:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494028800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762261.548852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762261.548913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff8eb64304ed80 x1876699494028416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762261.548939:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494028416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762261.548954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762261.548973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762261.548980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494028416:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762261.549027:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762261.549056:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762261.549073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff8eb64304d180 x1876699494028544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762261.549087:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494028544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762261.549097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762261.549101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762261.549105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028544:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789762261.549112:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028672:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789762261.549136:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762261.549149:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff8eb64304c000 x1876699494028672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762261.549162:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494028672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762261.549183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762261.549188:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762261.549192:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494028672:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762261.549199:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494028800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789762261.549223:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762261.549238:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762261/real 1789762261] req@ffff8eb64304e300 x1876699494028800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762261.549253:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494028800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762261.549263:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762261.549266:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789762261.549271:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494028800:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762262.252650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762263.276523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762264.300537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762265.324581:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762266.348577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762266.668519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.668527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762266.668531:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.668539:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2271 00000100:00080000:3.0:1789762266.668548:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.668554:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762266.668588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494028928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.668613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.668615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.668617:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.668620:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2271 00000100:00080000:3.0:1789762266.668622:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.668625:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762266.668634:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762266.668647:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494029056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762266.668648:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494028928:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762266.668662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.668664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.668666:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.668668:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2271 00000100:00080000:3.0:1789762266.668670:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.668673:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762266.668676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762266.668684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494029184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762266.668691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762266.668693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762266.668695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762266.668697:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2271 00000100:00080000:3.0:1789762266.668700:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762266.668702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762266.668705:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762266.668712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494029312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762266.668720:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762266.668757:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff8eb64514ad80 x1876699494028928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762266.668776:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494028928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762266.668786:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762266.668789:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762266.668793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494028928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762266.668815:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762266.668838:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762266.668850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff8eb645149180 x1876699494029056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762266.668861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494029056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762266.668868:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762266.668871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762266.668874:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029056:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762266.668880:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029184:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762266.668897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762266.668907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff8eb64514b480 x1876699494029184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762266.668919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494029184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762266.668940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762266.668943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762266.668947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029184:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762266.668953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494029312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762266.668971:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762266.668982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762266/real 1789762266] req@ffff8eb645148700 x1876699494029312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762266.668992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494029312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762266.669000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762266.669004:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762266.669007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494029312:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762267.372551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762268.396627:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762269.420560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762270.444546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762271.468571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762271.788471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.788479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762271.788483:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.788491:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2276 00000100:00080000:3.0:1789762271.788501:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.788507:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762271.788538:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494029440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.788561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.788563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762271.788565:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.788568:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2276 00000100:00080000:3.0:1789762271.788570:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.788573:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762271.788581:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.788590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494029568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762271.788597:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494029440:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762271.788601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.788605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762271.788606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.788609:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2276 00000100:00080000:3.0:1789762271.788611:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.788613:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762271.788617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762271.788625:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494029696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762271.788637:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762271.788639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762271.788640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762271.788643:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2276 00000100:00080000:3.0:1789762271.788647:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762271.788649:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762271.788652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762271.788659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494029824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762271.788690:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762271.788710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029696:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762271.788727:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494029824:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762272.492505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762273.516544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762274.540595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762274.860817:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762274.860893:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762274.860924:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762274.860966:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762274.860975:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762274.860984:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762274.861043:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762274] req@ffff8eb645148000 x1876699494029568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762274.861078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494029568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762274.861094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762274.861100:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762274.861105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762274.861122:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762274] req@ffff8eb645149880 x1876699494029696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762274.861138:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494029696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762274.861148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762274.861152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762274.861157:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494029696:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762274.861165:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762274] req@ffff8eb64514b100 x1876699494029824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762274.861178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494029824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762274.861196:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762274.861201:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762274.861206:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494029824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762274.861215:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762271/real 1789762274] req@ffff8eb64514bb80 x1876699494029440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762274.861231:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494029440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762274.861241:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762274.861244:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762274.861247:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494029440:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762275.564596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762276.588588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762276.908543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762276.908554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762276.908558:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762276.908569:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2281 00000100:00080000:3.0:1789762276.908581:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762276.908589:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762276.908639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494029952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762276.908678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762276.908682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762276.908684:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762276.908689:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2281 00000100:00080000:3.0:1789762276.908693:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762276.908698:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762276.908710:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762276.908726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494030080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762276.908737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762276.908740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762276.908742:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762276.908745:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2281 00000100:00080000:3.0:1789762276.908748:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762276.908750:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762276.908754:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762276.908759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494029952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762276.908767:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494030208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762276.908778:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762276.908782:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762276.908784:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762276.908789:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2281 00000100:00080000:3.0:1789762276.908792:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762276.908798:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762276.908803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762276.908816:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494030336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762276.908865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762276.908920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762276/real 1789762276] req@ffff8eb64514b800 x1876699494029952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762276.908943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494029952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762276.908955:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762276.908960:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762276.908964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494029952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762276.908977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762276.909004:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762276.909020:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762276/real 1789762276] req@ffff8eb64514a300 x1876699494030080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762276.909033:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494030080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762276.909042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762276.909046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762276.909050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030080:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762276.909083:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030208:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762276.909107:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762276.909120:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762276/real 1789762276] req@ffff8eb645148e00 x1876699494030208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762276.909131:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494030208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762276.909139:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762276.909143:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762276.909147:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030208:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762276.909153:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494030336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762276.909173:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762276.909186:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762276/real 1789762276] req@ffff8eb645148380 x1876699494030336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762276.909198:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494030336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762276.909207:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762276.909210:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762276.909213:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494030336:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762277.612544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762278.636610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762279.660586:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762280.684606:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762281.708623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762282.028472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762282.028480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762282.028484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762282.028492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2286 00000100:00080000:2.0:1789762282.028501:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762282.028507:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762282.028538:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494030464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762282.028559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762282.028562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762282.028564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762282.028566:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2286 00000100:00080000:2.0:1789762282.028569:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762282.028572:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762282.028579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762282.028588:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494030592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762282.028593:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494030464:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762282.028603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762282.028606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762282.028607:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762282.028610:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2286 00000100:00080000:2.0:1789762282.028613:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762282.028616:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762282.028619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762282.028629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494030720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762282.028636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762282.028639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762282.028640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762282.028643:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2286 00000100:00080000:2.0:1789762282.028645:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762282.028648:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762282.028651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762282.028655:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762282.028661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494030848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762282.028712:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff8eb650f80a80 x1876699494030464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762282.028726:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494030464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762282.028733:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762282.028736:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762282.028739:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494030464:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762282.028756:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762282.028770:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762282.028778:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff8eb650f80000 x1876699494030592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762282.028790:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494030592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762282.028821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762282.028823:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762282.028825:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030592:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762282.028829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030720:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762282.028842:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762282.028849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff8eb650f80380 x1876699494030720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762282.028857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494030720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762282.028862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762282.028864:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762282.028866:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494030720:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762282.028869:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494030848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762282.028878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762282.028883:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762282/real 1789762282] req@ffff8eb650f81500 x1876699494030848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762282.028890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494030848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762282.028899:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762282.028901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762282.028903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494030848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762282.732527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762283.756569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762284.780558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762285.804525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762286.828617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762287.148516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762287.148523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762287.148527:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762287.148536:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2291 00000100:00080000:2.0:1789762287.148574:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762287.148582:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762287.148622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494030976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762287.148651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762287.148655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762287.148657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762287.148662:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2291 00000100:00080000:2.0:1789762287.148665:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762287.148668:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762287.148678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762287.148689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494031104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762287.148709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762287.148712:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762287.148714:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762287.148717:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2291 00000100:00080000:2.0:1789762287.148720:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762287.148723:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762287.148727:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762287.148735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494031232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762287.148746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762287.148751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762287.148753:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762287.148757:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2291 00000100:00080000:2.0:1789762287.148760:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762287.148764:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762287.148787:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762287.148794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494030976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762287.148833:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494031360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762287.148910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762287.148948:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff8eb650f81500 x1876699494030976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762287.148982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494030976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762287.148993:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762287.148996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762287.149001:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494030976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762287.149008:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762287.149027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762287.149037:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff8eb650f80380 x1876699494031104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762287.149048:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494031104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762287.149061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762287.149064:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762287.149067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762287.149071:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031232:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762287.149086:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762287.149094:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff8eb650f80000 x1876699494031232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762287.149104:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494031232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762287.149111:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762287.149130:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762287.149134:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762287.149174:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494031360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762287.149194:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762287.149209:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762287/real 1789762287] req@ffff8eb650f80a80 x1876699494031360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762287.149220:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494031360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762287.149228:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762287.149231:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762287.149235:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494031360:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762287.852571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762288.876553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762289.900569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762290.924547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762291.948597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762292.268553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762292.268565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762292.268570:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762292.268582:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2296 00000100:00080000:3.0:1789762292.268595:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762292.268603:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762292.268647:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494031488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762292.268674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762292.268676:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762292.268678:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762292.268681:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2296 00000100:00080000:3.0:1789762292.268684:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762292.268686:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762292.268696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762292.268705:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494031616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762292.268712:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762292.268715:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762292.268716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762292.268718:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2296 00000100:00080000:3.0:1789762292.268722:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762292.268725:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762292.268756:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762292.268769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494031744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762292.268779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762292.268783:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494031488:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762292.268784:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762292.268787:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762292.268795:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2296 00000100:00080000:3.0:1789762292.268798:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762292.268801:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762292.268804:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762292.268814:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494031872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762292.268867:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762292.268911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762292] req@ffff8eb645149500 x1876699494031488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762292.268926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494031488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762292.268936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762292.268939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762292.268943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494031488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762292.268948:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762292.268963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762292.268971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762292] req@ffff8eb645148a80 x1876699494031616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762292.268979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494031616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762292.268984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762292.268986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762292.268988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031616:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762292.269012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031744:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762292.269024:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762292.269031:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762292] req@ffff8eb64514a680 x1876699494031744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762292.269038:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494031744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762292.269053:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762292.269055:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762292.269057:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494031744:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762292.269060:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494031872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762292.269071:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762292.269077:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762292/real 1789762292] req@ffff8eb645149c00 x1876699494031872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762292.269084:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494031872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762292.269089:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762292.269091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762292.269092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494031872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762292.972584:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762293.996523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762295.020570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762296.044553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762297.068547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762297.388537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.388546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762297.388550:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.388559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2301 00000100:00080000:1.0:1789762297.388569:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.388576:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762297.388611:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494032000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.388636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.388639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762297.388640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.388644:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2301 00000100:00080000:1.0:1789762297.388646:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.388649:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762297.388657:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762297.388667:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494032128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.388676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.388682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762297.388683:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494032000:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762297.388685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.388712:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2301 00000100:00080000:1.0:1789762297.388715:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.388718:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762297.388723:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762297.388736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494032256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762297.388750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762297.388753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762297.388755:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762297.388758:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2301 00000100:00080000:1.0:1789762297.388762:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762297.388765:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762297.388768:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762297.388777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762297.388781:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494032384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762297.388828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff8eb65016ce00 x1876699494032000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762297.388844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494032000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762297.388852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762297.388856:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762297.388859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494032000:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762297.388879:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762297.388898:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762297.388907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff8eb65016c700 x1876699494032128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762297.388917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494032128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762297.388923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762297.388926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762297.388928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762297.388932:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032256:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762297.388945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762297.388951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff8eb65016df80 x1876699494032256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762297.388966:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494032256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762297.388972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762297.388974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762297.388976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032256:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762297.388980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494032384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762297.388991:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762297.389006:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762297/real 1789762297] req@ffff8eb65016dc00 x1876699494032384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762297.389015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494032384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762297.389021:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762297.389023:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762297.389026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494032384:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762298.092600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762299.116554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762300.140544:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762301.164565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762302.188535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762302.508535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762302.508543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762302.508546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762302.508554:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2307 00000100:00080000:3.0:1789762302.508563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762302.508569:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762302.508608:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494032512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762302.508634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762302.508637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762302.508639:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762302.508643:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2307 00000100:00080000:3.0:1789762302.508646:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762302.508649:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762302.508668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762302.508679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494032640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762302.508686:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494032512:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762302.508690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762302.508694:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762302.508695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762302.508704:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2307 00000100:00080000:3.0:1789762302.508710:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762302.508713:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762302.508717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762302.508726:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494032768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762302.508733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762302.508736:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762302.508737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762302.508740:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2307 00000100:00080000:3.0:1789762302.508742:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762302.508745:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762302.508751:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762302.508751:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762302.508762:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494032896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762302.508800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff8eb645149c00 x1876699494032512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762302.508828:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494032512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762302.508840:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762302.508845:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762302.508850:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494032512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762302.508873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762302.508900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762302.508915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff8eb64514a680 x1876699494032640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762302.508928:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494032640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762302.508937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762302.508940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762302.508942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762302.508947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032768:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762302.508964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762302.508972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff8eb645148a80 x1876699494032768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762302.508980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494032768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762302.508989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762302.508991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762302.508994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494032768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762302.508997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494032896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762302.509007:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762302.509013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762302/real 1789762302] req@ffff8eb645149500 x1876699494032896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762302.509019:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494032896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762302.509025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762302.509026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762302.509028:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494032896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762303.212551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762304.236563:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762305.260544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762306.284578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762307.308532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762307.628500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762307.628509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762307.628513:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762307.628520:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2312 00000100:00080000:3.0:1789762307.628530:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762307.628536:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762307.628590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494033024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762307.628612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762307.628637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762307.628638:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762307.628641:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2312 00000100:00080000:3.0:1789762307.628644:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762307.628646:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762307.628654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762307.628663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494033152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762307.628676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762307.628691:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762307.628692:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762307.628698:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2312 00000100:00100000:0.0:1789762307.628700:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494033024:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762307.628701:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762307.628707:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762307.628712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762307.628720:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494033280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762307.628728:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762307.628730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762307.628732:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762307.628735:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2312 00000100:00080000:3.0:1789762307.628737:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762307.628740:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762307.628743:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762307.628749:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494033408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762307.628769:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762307.628812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff8eb645148380 x1876699494033024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762307.628826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494033024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762307.628838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762307.628841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762307.628844:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494033024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762307.628849:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762307.628866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762307.628875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff8eb645148e00 x1876699494033152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762307.628892:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494033152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762307.628897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762307.628899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762307.628902:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762307.628920:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033280:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762307.628930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762307.628937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff8eb64514a300 x1876699494033280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762307.628944:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494033280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762307.628949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762307.628951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762307.628953:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762307.628957:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494033408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762307.628966:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762307.628972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762307/real 1789762307] req@ffff8eb64514b800 x1876699494033408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762307.628979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494033408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762307.628984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762307.628985:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762307.628987:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494033408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762308.332693:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762309.356686:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762310.380566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762311.404525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762312.428660:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762312.748556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762312.748570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762312.748612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762312.748636:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2317 00000100:00080000:3.0:1789762312.748650:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762312.748661:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762312.748709:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494033536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762312.748761:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762312.748768:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762312.748771:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762312.748779:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2317 00000100:00080000:3.0:1789762312.748786:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762312.748822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494033536:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762312.748847:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762312.748864:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762312.748887:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494033664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762312.748912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762312.748921:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762312.748922:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762312.748930:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762312.748939:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2317 00000100:00080000:3.0:1789762312.748948:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762312.748955:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762312.748965:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762312.748971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff8eb64514bb80 x1876699494033536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762312.748992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494033536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.749004:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494033792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762312.749018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762312.749029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762312.749037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494033536:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762312.749037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762312.749051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762312.749056:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762312.749070:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033664:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789762312.749075:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2317 00000100:00080000:3.0:1789762312.749085:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762312.749093:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762312.749101:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762312.749102:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762312.749118:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff8eb64514b100 x1876699494033664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762312.749129:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494033920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762312.749140:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494033664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762312.749160:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762312.749166:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762312.749181:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762312.749191:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033792:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762312.749224:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762312.749242:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff8eb645149880 x1876699494033792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762312.749284:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494033792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762312.749295:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762312.749298:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762312.749301:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494033792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762312.749317:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494033920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762312.749349:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762312.749360:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762312/real 1789762312] req@ffff8eb645148000 x1876699494033920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762312.749377:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494033920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762312.749390:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762312.749426:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762312.749430:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494033920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762313.452650:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762314.476707:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762315.500547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762316.524530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762317.548599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762317.868515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762317.868526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762317.868530:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762317.868555:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2322 00000100:00080000:3.0:1789762317.868570:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762317.868580:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762317.868641:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494034048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762317.868668:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762317.868671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762317.868673:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762317.868676:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2322 00000100:00080000:3.0:1789762317.868679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762317.868682:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762317.868692:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762317.868703:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494034176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762317.868712:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762317.868715:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762317.868716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762317.868719:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2322 00000100:00080000:3.0:1789762317.868721:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762317.868723:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762317.868726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762317.868735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494034304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762317.868745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762317.868755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789762317.868757:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494034048:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762317.868758:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762317.868771:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2322 00000100:00080000:3.0:1789762317.868774:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762317.868780:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762317.868786:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762317.868802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494034432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762317.868869:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762317.868935:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762317/real 1789762317] req@ffff8eb645148700 x1876699494034048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762317.868955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494034048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762317.868968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762317.868974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762317.868979:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494034048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762317.869006:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762317.869034:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762317.869049:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762317/real 1789762317] req@ffff8eb64514b480 x1876699494034176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762317.869092:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494034176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762317.869104:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762317.869109:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762317.869114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762317.869170:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762317.869220:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762317.869232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762317/real 1789762317] req@ffff8eb645149180 x1876699494034304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762317.869246:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494034304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762317.869254:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762317.869258:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762317.869262:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762317.869269:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494034432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762317.869287:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762317.869313:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762317/real 1789762317] req@ffff8eb64514ad80 x1876699494034432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762317.869325:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494034432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762317.869340:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762317.869343:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762317.869347:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494034432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762318.572569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762319.596627:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762320.620621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762321.644536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762322.668531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762322.988531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762322.988572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762322.988576:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762322.988586:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2327 00000100:00080000:1.0:1789762322.988597:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762322.988604:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762322.988650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494034560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762322.988678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762322.988682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762322.988684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762322.988689:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2327 00000100:00080000:1.0:1789762322.988693:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762322.988697:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762322.988708:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762322.988720:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494034560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762322.988723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494034688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762322.988734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762322.988737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762322.988738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762322.988741:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2327 00000100:00080000:1.0:1789762322.988744:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762322.988747:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762322.988751:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762322.988759:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494034816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762322.988766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762322.988769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762322.988770:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762322.988773:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2327 00000800:00000100:0.0:1789762322.988778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762322.988779:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762322.988784:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762322.988788:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762322.988796:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494034944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762322.988821:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762322/real 1789762322] req@ffff8eb65016f480 x1876699494034560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762322.988843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494034560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762322.988855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762322.988859:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762322.988863:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494034560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762322.988888:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762322.988913:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762322.988928:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762322/real 1789762322] req@ffff8eb65016ed80 x1876699494034688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762322.988942:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494034688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762322.988953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762322.988956:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762322.988958:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762322.988963:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034816:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762322.988979:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762322.988986:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762322/real 1789762322] req@ffff8eb65016d180 x1876699494034816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762322.988994:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494034816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762322.988999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762322.989001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762322.989003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494034816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762322.989006:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494034944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762322.989016:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762322.989022:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762322/real 1789762322] req@ffff8eb65016c000 x1876699494034944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762322.989029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494034944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762322.989033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762322.989035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762322.989037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494034944:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762323.692551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762324.716522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762325.740577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762326.764519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762327.788589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762328.108529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.108536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762328.108540:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.108547:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2332 00000100:00080000:3.0:1789762328.108556:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.108562:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762328.108594:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494035072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.108614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.108617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.108618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.108621:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2332 00000100:00080000:3.0:1789762328.108623:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.108626:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762328.108634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762328.108642:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494035200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.108650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762328.108655:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494035072:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762328.108655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.108659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.108665:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2332 00000100:00080000:3.0:1789762328.108668:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.108670:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762328.108673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762328.108682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494035328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762328.108691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762328.108695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762328.108697:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762328.108701:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2332 00000100:00080000:3.0:1789762328.108706:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762328.108710:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762328.108715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762328.108720:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762328.108737:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494035456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762328.108750:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff8eb64514ad80 x1876699494035072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762328.108764:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494035072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762328.108772:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762328.108775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762328.108778:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494035072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762328.108799:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762328.108820:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762328.108833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff8eb645149180 x1876699494035200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762328.108847:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494035200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762328.108853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762328.108855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762328.108857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762328.108863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762328.108878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762328.108885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff8eb64514b480 x1876699494035328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762328.108893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494035328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762328.108898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762328.108900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762328.108902:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762328.108905:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494035456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762328.108915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762328.108920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762328/real 1789762328] req@ffff8eb645148700 x1876699494035456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762328.108927:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494035456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762328.108932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762328.108934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762328.108936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494035456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762328.812550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762329.836633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762330.860558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762331.884543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762332.908626:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762333.228586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.228599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762333.228604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.228617:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2337 00000100:00080000:3.0:1789762333.228632:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.228641:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762333.228696:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494035584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.228736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.228740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.228742:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.228747:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2337 00000100:00080000:3.0:1789762333.228766:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.228776:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762333.228792:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762333.228808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494035584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762333.228830:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494035712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.228855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762333.228859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.228861:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.228867:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2337 00000100:00080000:3.0:1789762333.228871:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.228877:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762333.228895:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762333.228911:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494035840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762333.228922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762333.228932:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762333.228938:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762333.228941:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762333.228948:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2337 00000100:00080000:3.0:1789762333.228955:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762333.228959:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762333.228964:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762333.228979:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494035968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762333.228990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff8eb645148000 x1876699494035584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.229012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494035584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.229074:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.229078:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762333.229083:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494035584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762333.229107:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.229127:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762333.229136:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff8eb645149880 x1876699494035712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.229144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494035712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.229150:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.229153:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762333.229171:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762333.229176:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762333.229189:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762333.229195:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff8eb64514b100 x1876699494035840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.229207:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494035840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.229212:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.229214:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762333.229215:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494035840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762333.229219:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494035968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762333.229229:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762333.229235:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762333/real 1789762333] req@ffff8eb64514bb80 x1876699494035968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762333.229241:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494035968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762333.229246:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762333.229248:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762333.229250:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494035968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762333.932621:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762334.956562:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762335.980605:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762337.004497:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762338.028515:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762338.348514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762338.348526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762338.348530:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762338.348541:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2342 00000100:00080000:3.0:1789762338.348553:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762338.348560:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762338.348605:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494036096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762338.348633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762338.348636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762338.348638:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762338.348641:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2342 00000100:00080000:3.0:1789762338.348643:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762338.348646:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762338.348656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762338.348670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494036224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762338.348685:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494036096:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762338.348688:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762338.348701:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762338.348704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762338.348711:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2342 00000100:00080000:3.0:1789762338.348716:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762338.348723:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762338.348729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762338.348743:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494036352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762338.348752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762338.348755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762338.348757:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762338.348760:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2342 00000100:00080000:3.0:1789762338.348764:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762338.348767:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762338.348771:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762338.348780:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494036480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762338.348811:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762338.348838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762338.348860:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494036480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762339.052511:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762340.076555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762341.100523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762341.420656:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789762341.420699:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762341.420716:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762341.420766:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762341.420779:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762341.420789:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789762341.420801:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762341] req@ffff8eb64514a300 x1876699494036224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762341.420836:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494036224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762341.420848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762341.420853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762341.420857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762341.420865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762341] req@ffff8eb645148e00 x1876699494036352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762341.420879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494036352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762341.420895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762341.420898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762341.420901:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762341.420908:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762341] req@ffff8eb645148380 x1876699494036480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762341.420920:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494036480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762341.420929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762341.420933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762341.420937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494036480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762341.420943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762338/real 1789762341] req@ffff8eb64514b800 x1876699494036096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762341.420957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494036096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762341.420965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762341.420969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762341.420972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494036096:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789762342.124530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762343.148515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762343.468535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762343.468547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762343.468552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762343.468563:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2347 00000100:00080000:3.0:1789762343.468575:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762343.468584:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762343.468644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494036608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762343.468703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762343.468708:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762343.468710:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762343.468714:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2347 00000100:00080000:3.0:1789762343.468718:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762343.468723:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762343.468732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762343.468747:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494036736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762343.468759:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762343.468762:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494036608:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762343.468764:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762343.468768:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762343.468777:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2347 00000100:00080000:3.0:1789762343.468781:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762343.468784:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762343.468790:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762343.468805:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494036864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762343.468816:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762343.468820:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762343.468823:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762343.468827:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2347 00000100:00080000:3.0:1789762343.468831:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762343.468836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762343.468837:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762343.468846:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762343.468858:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494036992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762343.468893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff8eb645149500 x1876699494036608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762343.468914:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494036608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762343.468924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762343.468928:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762343.468932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494036608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762343.468957:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762343.468983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762343.468997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff8eb645148a80 x1876699494036736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762343.469010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494036736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762343.469019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762343.469023:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762343.469026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762343.469032:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036864:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762343.469054:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762343.469066:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff8eb64514a680 x1876699494036864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762343.469079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494036864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762343.469088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762343.469091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762343.469095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494036864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762343.469101:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494036992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762343.469123:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762343.469135:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762343/real 1789762343] req@ffff8eb645149c00 x1876699494036992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762343.469146:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494036992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762343.469154:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762343.469158:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762343.469160:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494036992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762344.172620:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762345.196597:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762346.220712:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762347.244563:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762348.268543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762348.588493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.588504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762348.588508:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.588518:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2353 00000100:00080000:1.0:1789762348.588529:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.588536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762348.588574:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494037120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.588604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.588607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.588609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.588615:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2353 00000100:00080000:1.0:1789762348.588619:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.588626:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762348.588639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762348.588663:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494037120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762348.588664:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494037248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.588691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.588696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.588698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.588703:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2353 00000100:00080000:1.0:1789762348.588708:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.588730:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762348.588738:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762348.588768:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762348.588795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494037376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762348.588810:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762348.588813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762348.588815:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762348.588820:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2353 00000100:00080000:1.0:1789762348.588825:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762348.588830:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762348.588836:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff8eb65016f100 x1876699494037120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762348.588839:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762348.588859:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494037504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762348.588867:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494037120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762348.588886:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762348.588894:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762348.588900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494037120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762348.588927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762348.588950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762348.588962:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff8eb65016ca80 x1876699494037248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762348.588975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494037248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762348.588983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762348.588987:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762348.588990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762348.588996:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762348.589029:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762348.589041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff8eb65016c000 x1876699494037376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762348.589073:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494037376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762348.589084:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762348.589088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762348.589092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762348.589099:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494037504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762348.589123:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762348.589135:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762348/real 1789762348] req@ffff8eb65016d180 x1876699494037504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762348.589149:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494037504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762348.589158:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762348.589162:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762348.589165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494037504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762349.292543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762350.316579:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762351.340545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762352.364565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762353.388594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762353.708508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.708519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762353.708523:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.708536:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2358 00000100:00080000:2.0:1789762353.708547:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.708555:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762353.708599:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494037632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.708624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.708627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762353.708630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.708634:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2358 00000100:00080000:2.0:1789762353.708637:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.708640:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762353.708651:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762353.708655:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494037632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762353.708664:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494037760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.708673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.708675:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762353.708677:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.708680:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2358 00000100:00080000:2.0:1789762353.708682:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.708685:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762353.708704:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762353.708713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494037888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762353.708720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762353.708723:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762353.708725:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762353.708727:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2358 00000100:00080000:2.0:1789762353.708730:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762353.708733:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762353.708736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762353.708744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494038016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762353.708750:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762353.708795:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff8eb650f81f80 x1876699494037632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762353.708810:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494037632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762353.708819:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762353.708822:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762353.708825:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494037632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762353.708848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762353.708863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762353.708871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff8eb650f83b80 x1876699494037760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762353.708879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494037760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762353.708885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762353.708887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762353.708890:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762353.708893:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037888:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762353.708906:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762353.708912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff8eb650f81180 x1876699494037888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762353.708919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494037888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762353.708925:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762353.708926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762353.708928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494037888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762353.708931:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494038016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762353.708942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762353.708948:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762353/real 1789762353] req@ffff8eb650f83800 x1876699494038016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762353.708955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494038016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762353.708960:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762353.708962:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762353.708963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494038016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762354.412523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762355.436710:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762356.460614:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762357.484738:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762358.508517:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762358.828548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.828563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762358.828570:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.828598:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2363 00000100:00080000:2.0:1789762358.828616:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.828627:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762358.828702:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494038144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.828743:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.828765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.828770:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.828781:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2363 00000100:00080000:2.0:1789762358.828788:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.828797:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762358.828816:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762358.828818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494038144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762358.828884:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494038272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762358.828899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762358.828905:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.828925:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762358.828935:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762358.828942:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2363 00000100:00080000:2.0:1789762358.828950:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.828961:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762358.828972:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762358.828987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762358] req@ffff8eb650f82a00 x1876699494038144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762358.829017:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494038400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762358.829023:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494038144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762358.829096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762358.829107:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:2.0:1789762358.829108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762358.829114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494038144:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762358.829126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762358.829130:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762358.829139:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2363 00000100:00080000:2.0:1789762358.829144:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762358.829153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762358.829154:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038272:192.168.203.160@tcp:38:kworker.0 00000008:00100000:2.0:1789762358.829175:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762358.829199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494038528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762358.829213:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762358.829250:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762358] req@ffff8eb650f83480 x1876699494038272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762358.829268:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494038272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762358.829286:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762358.829290:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762358.829296:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762358.829303:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762358.829321:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762358.829328:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762358] req@ffff8eb650f80e00 x1876699494038400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762358.829336:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494038400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762358.829346:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762358.829348:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762358.829350:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762358.829361:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494038528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762358.829372:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762358.829378:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762358/real 1789762358] req@ffff8eb650f80700 x1876699494038528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762358.829386:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494038528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762358.829391:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762358.829424:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762358.829426:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494038528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762359.532573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762360.556548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762361.580563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762362.604545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762363.628548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762363.948580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762363.948591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762363.948596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762363.948609:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2368 00000100:00080000:3.0:1789762363.948641:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762363.948650:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762363.948714:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494038656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762363.948777:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762363.948782:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762363.948784:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762363.948788:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2368 00000100:00080000:3.0:1789762363.948791:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762363.948795:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762363.948805:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762363.948855:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494038784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762363.948870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762363.948875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762363.948886:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762363.948892:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2368 00000100:00080000:3.0:1789762363.948897:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762363.948903:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762363.948909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494038656:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762363.948921:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762363.948939:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494038912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762363.948961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762363.948964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762363.948966:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762363.948971:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2368 00000100:00080000:3.0:1789762363.948976:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762363.948980:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762363.948984:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762363.948994:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494039040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762363.949035:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762363.949115:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762363/real 1789762363] req@ffff8eb645149c00 x1876699494038656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762363.949134:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494038656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762363.949150:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762363.949153:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762363.949168:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494038656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762363.949174:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762363.949214:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762363.949243:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762363/real 1789762363] req@ffff8eb64514a680 x1876699494038784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762363.949254:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494038784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762363.949262:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762363.949264:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762363.949267:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762363.949306:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038912:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762363.949323:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762363.949331:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762363/real 1789762363] req@ffff8eb645148a80 x1876699494038912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762363.949340:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494038912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762363.949363:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762363.949364:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762363.949366:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494038912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762363.949371:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494039040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762363.949383:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762363.949389:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762363/real 1789762363] req@ffff8eb645149500 x1876699494039040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762363.949397:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494039040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762363.949406:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762363.949460:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762363.949465:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494039040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762364.652583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762365.676525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762366.700692:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762367.724539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762368.748656:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762369.068594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762369.068608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762369.068613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762369.068626:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2373 00000100:00080000:1.0:1789762369.068641:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762369.068650:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762369.068725:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494039168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762369.068769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762369.068773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762369.068776:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762369.068781:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2373 00000100:00080000:1.0:1789762369.068785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762369.068806:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762369.068819:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762369.068833:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494039296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762369.068844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762369.068847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762369.068850:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762369.068853:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2373 00000100:00080000:1.0:1789762369.068856:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762369.068861:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762369.068868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494039168:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762369.068869:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762369.068885:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494039424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762369.068895:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762369.068898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762369.068900:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762369.068904:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2373 00000100:00080000:1.0:1789762369.068907:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762369.068910:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762369.068915:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762369.068925:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494039552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762369.068963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762369.069013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff8eb65016ed80 x1876699494039168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762369.069032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494039168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762369.069043:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762369.069047:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762369.069051:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494039168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762369.069060:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762369.069081:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762369.069101:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff8eb65016f480 x1876699494039296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762369.069113:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494039296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762369.069120:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762369.069123:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762369.069127:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762369.069170:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762369.069189:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762369.069206:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff8eb65016dc00 x1876699494039424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762369.069217:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494039424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762369.069233:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762369.069236:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762369.069239:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762369.069244:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494039552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762369.069259:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762369.069268:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762369/real 1789762369] req@ffff8eb65016df80 x1876699494039552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762369.069279:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494039552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762369.069285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762369.069288:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762369.069290:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494039552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762369.772574:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762370.796598:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762371.820696:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762372.844563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762373.868502:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762374.188578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762374.188586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762374.188590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762374.188599:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2378 00000100:00080000:2.0:1789762374.188609:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762374.188615:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762374.188647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494039680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762374.188688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762374.188691:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762374.188694:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762374.188698:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2378 00000100:00080000:2.0:1789762374.188701:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762374.188705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762374.188714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762374.188726:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494039808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762374.188736:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762374.188739:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762374.188742:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762374.188763:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2378 00000100:00080000:2.0:1789762374.188766:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762374.188769:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762374.188770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494039680:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789762374.188773:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762374.188786:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494039936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762374.188808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762374.188811:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762374.188812:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762374.188815:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2378 00000100:00080000:2.0:1789762374.188817:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762374.188819:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762374.188823:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762374.188831:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494040064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762374.188876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762374.188935:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff8eb650f82680 x1876699494039680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762374.188953:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494039680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762374.188964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762374.188968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762374.188972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494039680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762374.188995:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762374.189026:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762374.189052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff8eb650f82d80 x1876699494039808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762374.189066:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494039808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762374.189080:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762374.189082:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762374.189084:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762374.189088:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762374.189100:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762374.189107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff8eb650f81880 x1876699494039936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762374.189115:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494039936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762374.189137:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762374.189139:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762374.189141:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494039936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762374.189145:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494040064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762374.189161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762374.189175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762374/real 1789762374] req@ffff8eb650f81c00 x1876699494040064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762374.189182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494040064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762374.189192:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762374.189195:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762374.189197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494040064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762374.892510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762375.916521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762376.940630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762377.964577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762378.988564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762379.308540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762379.308553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762379.308559:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762379.308572:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2383 00000100:00080000:2.0:1789762379.308587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762379.308597:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762379.308647:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494040192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762379.308683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762379.308686:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762379.308707:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762379.308712:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2383 00000100:00080000:2.0:1789762379.308717:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762379.308721:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762379.308732:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762379.308749:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494040320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762379.308762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762379.308767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762379.308772:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762379.308774:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494040192:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762379.308780:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2383 00000100:00080000:2.0:1789762379.308790:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762379.308796:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762379.308802:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762379.308817:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494040448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762379.308844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762379.308847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762379.308849:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762379.308854:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2383 00000100:00080000:2.0:1789762379.308858:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762379.308862:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762379.308867:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762379.308878:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494040576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762379.308890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762379.308943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff8eb650f81c00 x1876699494040192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762379.308962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494040192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762379.308972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762379.308976:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762379.308980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494040192:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762379.309015:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762379.309047:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762379.309061:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff8eb650f81880 x1876699494040320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762379.309107:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494040320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762379.309122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762379.309126:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762379.309129:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762379.309136:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040448:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762379.309158:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762379.309169:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff8eb650f82d80 x1876699494040448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762379.309180:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494040448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762379.309187:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762379.309190:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762379.309193:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762379.309198:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494040576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762379.309212:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762379.309220:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762379/real 1789762379] req@ffff8eb650f82680 x1876699494040576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762379.309231:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494040576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762379.309238:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762379.309241:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762379.309244:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494040576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762380.012581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762381.036591:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762382.060516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762383.084542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762384.108512:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762384.428468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762384.428479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762384.428484:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762384.428494:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2388 00000100:00080000:2.0:1789762384.428505:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762384.428512:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762384.428548:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494040704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762384.428582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762384.428586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762384.428588:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762384.428594:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2388 00000100:00100000:0.0:1789762384.428600:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494040704:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762384.428601:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762384.428610:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762384.428620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762384.428635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494040832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762384.428645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762384.428648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762384.428650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762384.428654:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2388 00000100:00080000:2.0:1789762384.428657:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762384.428659:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762384.428664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762384.428674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494040960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762384.428679:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762384.428691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762384.428694:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762384.428697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762384.428702:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2388 00000100:00080000:2.0:1789762384.428705:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762384.428708:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762384.428712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762384.428724:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff8eb650f80700 x1876699494040704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762384.428725:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494041088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762384.428778:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494040704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762384.428790:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762384.428793:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762384.428805:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494040704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762384.428827:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762384.428845:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762384.428855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff8eb650f80e00 x1876699494040832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762384.428866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494040832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762384.428873:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762384.428875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762384.428878:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762384.428884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762384.428896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762384.428903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff8eb650f83480 x1876699494040960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762384.428912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494040960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762384.428919:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762384.428921:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762384.428923:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494040960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762384.428927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494041088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762384.428939:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762384.428946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762384/real 1789762384] req@ffff8eb650f82a00 x1876699494041088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762384.428955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494041088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762384.428961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762384.428963:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762384.428966:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494041088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762385.132524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762386.156589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762387.180541:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762388.204557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762389.228570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762389.548537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.548551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762389.548564:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.548579:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2394 00000100:00080000:3.0:1789762389.548593:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.548602:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762389.548653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494041216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.548688:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.548693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.548695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.548700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2394 00000100:00080000:3.0:1789762389.548704:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.548709:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762389.548721:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762389.548746:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494041344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.548759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.548763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.548766:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.548772:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2394 00000100:00080000:3.0:1789762389.548778:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.548783:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762389.548791:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762389.548798:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494041216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762389.548810:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494041472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762389.548826:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762389.548831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762389.548833:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762389.548838:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2394 00000100:00080000:3.0:1789762389.548843:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762389.548848:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762389.548853:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762389.548867:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494041600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762389.548878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762389.548926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff8eb64514b800 x1876699494041216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762389.548944:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494041216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762389.548953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762389.548957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762389.548960:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494041216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762389.548966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762389.548998:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762389.549025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff8eb645148380 x1876699494041344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762389.549034:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494041344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762389.549040:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762389.549042:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762389.549044:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762389.549064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041472:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762389.549078:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762389.549085:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff8eb645148e00 x1876699494041472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762389.549097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494041472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762389.549102:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762389.549104:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762389.549106:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762389.549110:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494041600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762389.549120:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762389.549126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762389/real 1789762389] req@ffff8eb64514a300 x1876699494041600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762389.549133:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494041600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762389.549148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762389.549151:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762389.549153:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494041600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762390.252506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762391.276509:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762392.300552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762393.324524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762394.348563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762394.668536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762394.668546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762394.668551:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762394.668575:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2399 00000100:00080000:1.0:1789762394.668587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762394.668620:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762394.668660:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494041728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762394.668690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762394.668693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762394.668695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762394.668699:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2399 00000100:00080000:1.0:1789762394.668703:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762394.668707:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762394.668717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762394.668734:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494041856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762394.668749:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494041728:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762394.668754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762394.668759:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762394.668762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762394.668775:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2399 00000100:00080000:1.0:1789762394.668781:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762394.668785:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762394.668800:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762394.668821:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494041984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762394.668844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762394.668849:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762394.668852:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762394.668857:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2399 00000100:00080000:1.0:1789762394.668861:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762394.668868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762394.668870:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762394.668877:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762394.668891:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494042112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762394.668931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff8eb65016c700 x1876699494041728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762394.668962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494041728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762394.668989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762394.669011:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762394.669016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494041728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762394.669046:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762394.669074:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762394.669089:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff8eb65016ce00 x1876699494041856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762394.669104:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494041856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762394.669113:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762394.669117:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762394.669121:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762394.669129:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762394.669151:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762394.669163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff8eb65016c380 x1876699494041984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762394.669173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494041984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762394.669183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762394.669186:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762394.669188:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494041984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762394.669193:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494042112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762394.669207:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762394.669214:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762394/real 1789762394] req@ffff8eb65016fb80 x1876699494042112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762394.669222:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494042112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762394.669228:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762394.669230:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762394.669232:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494042112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762395.372525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762396.396553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762397.420542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762398.444560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762399.468514:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762399.788496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762399.788511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762399.788514:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762399.788522:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2404 00000100:00080000:1.0:1789762399.788531:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762399.788537:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762399.788572:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494042240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762399.788598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762399.788602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762399.788604:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762399.788609:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2404 00000100:00080000:1.0:1789762399.788612:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762399.788616:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762399.788624:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762399.788635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494042368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762399.788644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762399.788647:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494042240:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762399.788647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762399.788651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762399.788658:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2404 00000100:00080000:1.0:1789762399.788660:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762399.788662:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762399.788665:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762399.788674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494042496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762399.788681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762399.788684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762399.788685:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762399.788688:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2404 00000100:00080000:1.0:1789762399.788690:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762399.788692:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762399.788695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762399.788702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494042624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762399.788722:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762399.788764:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff8eb65016d500 x1876699494042240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762399.788785:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494042240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762399.788805:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762399.788809:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762399.788814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494042240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762399.788837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762399.788878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762399.788895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff8eb65016e300 x1876699494042368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762399.788909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494042368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762399.788918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762399.788922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762399.788941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762399.788949:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762399.788990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762399.789005:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff8eb65016f800 x1876699494042496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762399.789020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494042496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762399.789030:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762399.789034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762399.789039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762399.789046:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494042624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762399.789071:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762399.789084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762399/real 1789762399] req@ffff8eb65016e680 x1876699494042624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762399.789097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494042624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762399.789107:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762399.789111:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762399.789115:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494042624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762400.492561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762401.516516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762402.540580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762403.564513:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762404.588517:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762404.908491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762404.908499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762404.908504:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762404.908513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2409 00000100:00080000:1.0:1789762404.908522:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762404.908529:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762404.908559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494042752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762404.908596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762404.908600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762404.908602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762404.908607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2409 00000100:00080000:1.0:1789762404.908611:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762404.908616:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762404.908626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762404.908641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494042880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762404.908653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762404.908657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762404.908659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762404.908663:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2409 00000100:00080000:1.0:1789762404.908670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762404.908680:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494042752:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762404.908685:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762404.908693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762404.908708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494043008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762404.908727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762404.908731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762404.908733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762404.908737:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2409 00000100:00080000:1.0:1789762404.908742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762404.908746:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762404.908764:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762404.908779:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494043136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762404.908783:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762404.908820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762404.908836:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494043136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762405.612482:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762406.636506:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762407.660580:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789762407.980727:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789762407.980775:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789762407.980805:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789762407.980849:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789762407.980858:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789762407.980910:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:2.0:1789762407.980938:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762404/real 1789762407] req@ffff8eb65016e680 x1876699494043008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762407.980967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494043008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762407.980980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762407.980986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762407.980991:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762407.980998:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762404/real 1789762407] req@ffff8eb65016f800 x1876699494043136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762407.981008:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494043136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762407.981013:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762407.981016:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762407.981018:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494043136:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762407.981025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762404/real 1789762407] req@ffff8eb65016ea00 x1876699494042752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762407.981035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494042752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762407.981042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762407.981046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762407.981049:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494042752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762407.981055:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762404/real 1789762407] req@ffff8eb65016d880 x1876699494042880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762407.981068:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494042880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762407.981075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762407.981079:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762407.981082:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494042880:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762408.684531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762409.708521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762410.028497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762410.028509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762410.028513:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762410.028526:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2414 00000100:00080000:1.0:1789762410.028539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762410.028549:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762410.028599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494043264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762410.028620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762410.028623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762410.028625:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762410.028627:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2414 00000100:00080000:1.0:1789762410.028630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762410.028633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762410.028643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762410.028652:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494043392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762410.028661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762410.028665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762410.028667:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762410.028667:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494043264:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762410.028672:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2414 00000100:00080000:1.0:1789762410.028676:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762410.028680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762410.028683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762410.028691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494043520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762410.028697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762410.028700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762410.028701:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762410.028705:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2414 00000100:00080000:1.0:1789762410.028709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762410.028713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762410.028718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762410.028732:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494043648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762410.028739:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762410.028788:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff8eb65016e300 x1876699494043264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762410.028810:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494043264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762410.028820:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762410.028824:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762410.028827:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494043264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762410.028847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762410.028887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762410.028917:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff8eb65016d500 x1876699494043392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762410.028931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494043392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762410.028940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762410.028943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762410.028947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762410.028953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762410.028976:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762410.028990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff8eb65016fb80 x1876699494043520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762410.029002:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494043520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762410.029011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762410.029014:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762410.029018:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762410.029024:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494043648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762410.029045:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762410.029058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762410/real 1789762410] req@ffff8eb65016c380 x1876699494043648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762410.029069:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494043648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762410.029077:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762410.029080:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762410.029083:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494043648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762410.732556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762411.756584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762412.780502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762413.804545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762414.828630:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762415.148511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762415.148538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762415.148543:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762415.148553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2419 00000100:00080000:3.0:1789762415.148564:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762415.148572:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762415.148612:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494043776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762415.148643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762415.148648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762415.148650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762415.148655:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2419 00000100:00080000:3.0:1789762415.148658:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762415.148663:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762415.148676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762415.148692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494043904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762415.148701:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494043776:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762415.148710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762415.148715:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762415.148717:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762415.148732:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2419 00000100:00080000:3.0:1789762415.148737:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762415.148742:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762415.148750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762415.148763:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494044032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762415.148775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762415.148780:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762415.148784:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762415.148787:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762415.148791:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2419 00000100:00080000:3.0:1789762415.148796:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762415.148799:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762415.148804:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762415.148825:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494044160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762415.148839:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff8eb64514bb80 x1876699494043776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762415.148858:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494043776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762415.148867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762415.148871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762415.148874:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494043776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762415.148898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762415.148915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762415.148924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff8eb64514b100 x1876699494043904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762415.148937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494043904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762415.148943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762415.148945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762415.148946:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494043904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762415.148950:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762415.148961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762415.148968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff8eb645149880 x1876699494044032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762415.148975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494044032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762415.148980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762415.148982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762415.148984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762415.148987:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494044160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762415.148997:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762415.149002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762415/real 1789762415] req@ffff8eb645148000 x1876699494044160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762415.149008:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494044160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762415.149017:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762415.149019:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762415.149021:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494044160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762415.852653:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762416.876557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762417.900596:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762418.924563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762419.948613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762420.268502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762420.268512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762420.268517:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762420.268528:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2424 00000100:00080000:2.0:1789762420.268540:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762420.268547:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762420.268592:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494044288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762420.268620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762420.268623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762420.268624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762420.268627:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2424 00000100:00080000:2.0:1789762420.268630:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762420.268632:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762420.268643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762420.268652:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494044416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762420.268660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762420.268663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762420.268666:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762420.268669:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2424 00000100:00080000:2.0:1789762420.268671:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762420.268674:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762420.268677:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762420.268685:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494044544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762420.268696:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494044288:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762420.268699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762420.268713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762420.268715:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762420.268721:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2424 00000100:00080000:2.0:1789762420.268725:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762420.268730:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762420.268736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762420.268751:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494044672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762420.268799:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762420.268846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff8eb650f83800 x1876699494044288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762420.268867:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494044288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762420.268880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762420.268884:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762420.268889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494044288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762420.268897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762420.268922:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762420.268937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff8eb650f81180 x1876699494044416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762420.268951:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494044416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762420.268961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762420.268965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762420.268969:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762420.268997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044544:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762420.269023:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762420.269036:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff8eb650f83b80 x1876699494044544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762420.269049:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494044544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762420.269059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762420.269063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762420.269067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762420.269074:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494044672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762420.269095:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762420.269106:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762420/real 1789762420] req@ffff8eb650f81f80 x1876699494044672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762420.269120:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494044672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762420.269129:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762420.269133:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762420.269137:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494044672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762420.972518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762421.996543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762423.020529:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762424.044659:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762425.068608:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762425.388584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.388596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762425.388602:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.388616:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2429 00000100:00080000:1.0:1789762425.388630:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.388664:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762425.388710:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494044800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.388789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.388795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762425.388797:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.388804:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2429 00000100:00080000:1.0:1789762425.388809:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.388815:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762425.388828:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762425.388849:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494044928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.388874:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.388884:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762425.388884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494044800:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762425.388891:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.388905:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2429 00000100:00080000:1.0:1789762425.388910:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.388914:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762425.388920:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762425.388938:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494045056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762425.388960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762425.388964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762425.388968:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762425.388973:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2429 00000100:00080000:1.0:1789762425.388977:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762425.388984:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:2.0:1789762425.388993:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762425.388997:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762425.389017:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494045184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762425.389072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762425] req@ffff8eb65016ce00 x1876699494044800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762425.389098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494044800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762425.389112:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762425.389118:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762425.389123:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494044800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762425.389151:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762425.389179:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762425.389194:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762425] req@ffff8eb65016c700 x1876699494044928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762425.389209:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494044928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762425.389220:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762425.389224:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762425.389228:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494044928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762425.389235:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762425.389265:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762425.389276:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762425] req@ffff8eb65016df80 x1876699494045056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762425.389290:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494045056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762425.389299:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762425.389303:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762425.389307:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762425.389313:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494045184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762425.389338:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762425.389349:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762425/real 1789762425] req@ffff8eb65016dc00 x1876699494045184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762425.389363:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494045184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762425.389372:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762425.389375:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762425.389390:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494045184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762426.092510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762427.116525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762428.140506:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762429.164562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762430.188517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762430.508488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762430.508502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762430.508506:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762430.508514:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2435 00000100:00080000:2.0:1789762430.508524:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762430.508530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762430.508562:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494045312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762430.508586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762430.508589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762430.508590:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762430.508593:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2435 00000100:00080000:2.0:1789762430.508595:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762430.508597:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762430.508605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762430.508615:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494045440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762430.508622:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762430.508625:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762430.508626:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762430.508628:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2435 00000100:00080000:2.0:1789762430.508631:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762430.508636:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494045312:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762430.508636:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762430.508640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762430.508650:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494045568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762430.508657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762430.508659:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762430.508661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762430.508663:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2435 00000100:00080000:2.0:1789762430.508665:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762430.508667:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762430.508669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762430.508678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494045696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762430.508717:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762430.508750:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff8eb650f80a80 x1876699494045312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762430.508763:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494045312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.508771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762430.508774:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762430.508777:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494045312:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762430.508782:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762430.508797:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762430.508805:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff8eb650f80000 x1876699494045440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762430.508812:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494045440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.508817:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762430.508819:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762430.508821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762430.508839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045568:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762430.508849:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762430.508855:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff8eb650f80380 x1876699494045568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762430.508863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494045568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.508869:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762430.508870:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762430.508872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762430.508876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494045696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762430.508885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762430.508890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762430/real 1789762430] req@ffff8eb650f81500 x1876699494045696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762430.508898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494045696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762430.508902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762430.508904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762430.508906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494045696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762431.212568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762432.236539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762433.260589:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762434.284603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762435.308612:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762435.628521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.628531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762435.628536:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.628546:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2440 00000100:00080000:0.0:1789762435.628557:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.628565:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762435.628605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494045824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.628637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.628641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.628643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.628647:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2440 00000100:00080000:0.0:1789762435.628650:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.628654:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762435.628680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762435.628698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494045952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762435.628742:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494045824:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762435.628745:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762435.628755:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.628758:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.628765:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2440 00000100:00080000:0.0:1789762435.628771:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.628778:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762435.628786:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762435.628805:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494046080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762435.628827:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762435.628828:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762435.628836:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762435.628839:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762435.628845:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2440 00000100:00080000:0.0:1789762435.628852:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762435.628857:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762435.628863:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762435.628877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff8eb64304f800 x1876699494045824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762435.628879:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494046208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762435.628943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494045824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.628962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.628981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762435.628985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494045824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762435.629012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.629039:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762435.629051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff8eb64304c380 x1876699494045952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.629065:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494045952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.629073:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.629076:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762435.629079:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494045952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762435.629084:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762435.629111:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762435.629120:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff8eb64304dc00 x1876699494046080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.629137:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494046080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.629148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.629153:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762435.629169:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762435.629178:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494046208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762435.629206:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762435.629219:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762435/real 1789762435] req@ffff8eb64304ce00 x1876699494046208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762435.629231:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494046208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762435.629263:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762435.629266:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762435.629269:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494046208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762436.332608:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762437.356579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762438.380571:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762439.404573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762440.428664:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762440.748612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762440.748644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762440.748651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762440.748680:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2445 00000100:00080000:2.0:1789762440.748693:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762440.748701:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762440.748747:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494046336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762440.748780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762440.748783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762440.748789:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762440.748794:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2445 00000100:00080000:2.0:1789762440.748797:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762440.748801:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762440.748810:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762440.748822:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494046464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762440.748832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762440.748835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762440.748837:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762440.748840:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2445 00000100:00080000:2.0:1789762440.748843:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762440.748847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494046336:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762440.748855:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762440.748862:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762440.748874:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494046592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762440.748889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762440.748892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762440.748893:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762440.748896:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2445 00000100:00080000:2.0:1789762440.748906:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762440.748911:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762440.748914:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762440.748927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762440.748928:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494046720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762440.748974:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff8eb650f81500 x1876699494046336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.748992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494046336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.749001:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.749005:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762440.749008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494046336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762440.749014:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.749032:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762440.749041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff8eb650f80380 x1876699494046464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.749051:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494046464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.749058:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.749060:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762440.749063:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046464:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762440.749084:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046592:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762440.749098:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762440.749105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff8eb650f80000 x1876699494046592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.749115:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494046592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.749121:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.749123:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762440.749126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046592:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762440.749130:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494046720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762440.749142:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762440.749149:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762440/real 1789762440] req@ffff8eb650f80a80 x1876699494046720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762440.749158:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494046720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762440.749164:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762440.749166:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762440.749168:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494046720:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762441.452632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762442.476511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762443.500518:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762444.524728:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762445.548659:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762445.868478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762445.868490:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762445.868495:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762445.868507:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2450 00000100:00080000:2.0:1789762445.868521:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762445.868530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762445.868573:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494046848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762445.868631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762445.868638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762445.868641:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762445.868649:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2450 00000100:00080000:2.0:1789762445.868657:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762445.868667:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762445.868684:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762445.868717:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494046976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762445.868721:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494046848:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762445.868748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762445.868753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762445.868756:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762445.868761:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2450 00000100:00080000:2.0:1789762445.868768:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762445.868772:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762445.868781:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762445.868797:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494047104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762445.868811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762445.868819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762445.868820:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762445.868823:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762445.868831:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2450 00000100:00080000:2.0:1789762445.868838:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762445.868843:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762445.868849:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762445.868863:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494047232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762445.868877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762445/real 1789762445] req@ffff8eb650f81f80 x1876699494046848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762445.868915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494046848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762445.868929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762445.868934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762445.868939:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494046848:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762445.868967:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762445.868992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762445.869007:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762445/real 1789762445] req@ffff8eb650f83b80 x1876699494046976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762445.869021:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494046976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762445.869031:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762445.869035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762445.869039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494046976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762445.869045:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047104:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762445.869063:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762445.869073:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762445/real 1789762445] req@ffff8eb650f81180 x1876699494047104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762445.869086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494047104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762445.869103:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762445.869106:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762445.869110:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762445.869116:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494047232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762445.869135:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762445.869145:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762445/real 1789762445] req@ffff8eb650f83800 x1876699494047232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762445.869158:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494047232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762445.869167:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762445.869170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762445.869174:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494047232:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762446.572613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762447.596608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762448.620627:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762449.644512:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762450.668675:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762450.988652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762450.988666:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762450.988671:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762450.988683:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2455 00000100:00080000:1.0:1789762450.988697:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762450.988723:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762450.988770:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494047360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762450.988813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762450.988819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762450.988821:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762450.988826:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2455 00000100:00080000:1.0:1789762450.988831:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762450.988836:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762450.988852:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762450.988879:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494047488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762450.988902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762450.988906:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494047360:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762450.988913:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762450.988921:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762450.988933:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2455 00000100:00080000:1.0:1789762450.988941:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762450.988984:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762450.988996:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762450.989021:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494047616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762450.989052:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762450.989076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762450.989084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762450.989089:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762450.989099:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2455 00000100:00080000:1.0:1789762450.989107:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762450.989117:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762450.989127:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762450/real 1789762450] req@ffff8eb65016f480 x1876699494047360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762450.989134:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762450.989164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494047744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762450.989179:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494047360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762450.989206:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762450.989232:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762450.989241:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494047360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762450.989281:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762450.989348:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762450.989392:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762450/real 1789762450] req@ffff8eb65016ed80 x1876699494047488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762450.989416:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494047488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762450.989432:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762450.989438:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762450.989442:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047488:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762450.989451:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047616:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762450.989538:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762450.989602:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762450/real 1789762450] req@ffff8eb65016d180 x1876699494047616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762450.989618:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494047616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762450.989629:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762450.989632:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762450.989636:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494047616:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762450.989643:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494047744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762450.989669:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762450.989680:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762450/real 1789762450] req@ffff8eb65016c000 x1876699494047744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762450.989713:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494047744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762450.989722:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762450.989726:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762450.989729:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494047744:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762451.692648:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762452.716591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762453.740644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762454.764574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762455.788595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762456.108526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.108540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762456.108546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.108559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2460 00000100:00080000:1.0:1789762456.108573:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.108583:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762456.108647:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494047872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762456.108680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.108684:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.108687:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.108692:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2460 00000100:00080000:1.0:1789762456.108696:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.108703:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762456.108722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762456.108728:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494047872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762456.108741:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494048000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762456.108756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762456.108760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.108763:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.108767:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2460 00000100:00080000:1.0:1789762456.108773:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.108777:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762456.108784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762456.108800:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494048128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762456.108812:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762456.108817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762456.108819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762456.108823:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762456.108829:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2460 00000100:00080000:1.0:1789762456.108834:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762456.108839:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762456.108846:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762456.108859:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494048256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762456.108881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff8eb65016ca80 x1876699494047872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.108903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494047872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.108917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.108922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762456.108926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494047872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762456.108957:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762456.108986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762456.109002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff8eb65016f100 x1876699494048000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.109024:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494048000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.109033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.109037:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762456.109041:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762456.109048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048128:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762456.109071:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762456.109084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff8eb65016c000 x1876699494048128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.109097:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494048128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.109107:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.109111:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762456.109114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762456.109121:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494048256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762456.109141:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762456.109153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762456/real 1789762456] req@ffff8eb65016d180 x1876699494048256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762456.109164:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494048256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762456.109179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762456.109183:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762456.109186:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494048256:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762456.812579:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762457.836551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762458.860648:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762459.884581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762460.908612:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762461.228607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762461.228620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762461.228626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762461.228639:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2465 00000100:00080000:0.0:1789762461.228652:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762461.228660:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762461.228704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494048384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762461.228739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762461.228744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762461.228746:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762461.228751:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2465 00000100:00080000:0.0:1789762461.228755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762461.228759:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762461.228770:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762461.228785:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494048512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762461.228792:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494048384:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762461.228814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762461.228818:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762461.228820:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762461.228825:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2465 00000100:00080000:0.0:1789762461.228829:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762461.228833:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762461.228838:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762461.228858:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494048640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762461.228881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762461.228888:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762461.228892:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762461.228899:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2465 00000100:00080000:0.0:1789762461.228906:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762461.228915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762461.228919:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762461.228932:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762461.228954:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494048768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762461.228989:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff8eb64304d500 x1876699494048384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762461.229015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494048384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762461.229028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762461.229032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762461.229037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494048384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762461.229067:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762461.229096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762461.229109:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff8eb64304d880 x1876699494048512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762461.229124:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494048512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762461.229133:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762461.229136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762461.229153:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762461.229163:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048640:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762461.229197:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762461.229217:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff8eb64304f480 x1876699494048640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762461.229255:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494048640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762461.229266:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762461.229269:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762461.229273:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494048640:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762461.229281:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494048768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762461.229308:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762461.229320:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762461/real 1789762461] req@ffff8eb64304e680 x1876699494048768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762461.229334:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494048768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762461.229343:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762461.229346:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762461.229349:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494048768:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762461.932578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762462.956561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762463.980639:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762465.004564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762466.028704:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762466.348526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762466.348534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762466.348539:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762466.348548:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2470 00000100:00080000:0.0:1789762466.348557:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762466.348563:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762466.348597:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494048896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762466.348619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762466.348622:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762466.348623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762466.348626:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2470 00000100:00080000:0.0:1789762466.348632:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762466.348635:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762466.348642:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762466.348652:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494049024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762466.348660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762466.348662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762466.348664:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762466.348666:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2470 00000100:00080000:0.0:1789762466.348668:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762466.348671:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762466.348675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762466.348680:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494048896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762466.348685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494049152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762466.348702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762466.348706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762466.348708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762466.348713:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2470 00000100:00080000:0.0:1789762466.348738:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762466.348744:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762466.348751:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762466.348751:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762466.348777:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494049280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762466.348810:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff8eb64304f100 x1876699494048896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762466.348844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494048896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762466.348858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762466.348862:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762466.348867:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494048896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762466.348875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762466.348901:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762466.348916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff8eb64304ca80 x1876699494049024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762466.348931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494049024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762466.348962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762466.348965:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762466.348967:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049024:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762466.348988:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049152:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762466.349011:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762466.349020:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff8eb64304ea00 x1876699494049152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762466.349029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494049152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762466.349035:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762466.349037:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762466.349039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049152:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762466.349043:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494049280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762466.349054:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762466.349068:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762466/real 1789762466] req@ffff8eb64304c700 x1876699494049280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762466.349076:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494049280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762466.349082:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762466.349084:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762466.349086:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494049280:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762467.052527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762468.076552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762469.100554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762470.124529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762471.148498:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762471.468503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762471.468511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762471.468515:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762471.468522:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2475 00000100:00080000:0.0:1789762471.468532:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762471.468538:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762471.468571:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494049408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762471.468593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762471.468596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762471.468597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762471.468600:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2475 00000100:00080000:0.0:1789762471.468603:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762471.468605:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762471.468613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762471.468622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494049536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762471.468629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762471.468632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762471.468633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762471.468636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2475 00000100:00080000:0.0:1789762471.468638:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762471.468642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762471.468644:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494049408:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789762471.468646:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762471.468656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494049664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762471.468664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762471.468667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762471.468668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762471.468671:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2475 00000100:00080000:0.0:1789762471.468674:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762471.468677:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762471.468680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762471.468687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494049792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762471.468734:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049536:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762471.468761:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762471.468778:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494049792:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762472.172578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762473.196594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762474.220525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762474.540708:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789762474.540760:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762474.540783:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762474.540832:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762474.540863:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762474.540876:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789762474.540899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762474] req@ffff8eb64304ca80 x1876699494049664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762474.540933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494049664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762474.540948:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762474.540955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762474.540960:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762474.540969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762474] req@ffff8eb64304f100 x1876699494049792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762474.540983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494049792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762474.540991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762474.540995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762474.540999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494049792:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762474.541011:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762474] req@ffff8eb64304c700 x1876699494049408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762474.541025:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494049408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762474.541032:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762474.541036:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762474.541039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494049408:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762474.541047:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762471/real 1789762474] req@ffff8eb64304ea00 x1876699494049536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762474.541060:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494049536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762474.541068:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762474.541072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762474.541076:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494049536:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762475.244629:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762476.268528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762476.588506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.588516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762476.588519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.588529:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2481 00000100:00080000:0.0:1789762476.588539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.588545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762476.588578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494049920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.588610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.588614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.588616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.588620:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2481 00000100:00080000:0.0:1789762476.588630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.588634:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762476.588643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762476.588655:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494050048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.588665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.588669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.588672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762476.588673:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494049920:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762476.588679:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2481 00000100:00080000:0.0:1789762476.588685:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.588688:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762476.588691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762476.588707:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494050176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762476.588715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762476.588718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762476.588719:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762476.588723:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2481 00000100:00080000:0.0:1789762476.588732:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762476.588736:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762476.588739:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762476.588748:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762476.588749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494050304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762476.588804:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff8eb64304e680 x1876699494049920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.588826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494049920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.588840:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.588860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762476.588865:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494049920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762476.588892:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.588918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762476.588932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff8eb64304f480 x1876699494050048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.588946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494050048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.588969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.588973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762476.588977:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050048:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762476.588984:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050176:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762476.589007:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762476.589019:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff8eb64304d880 x1876699494050176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.589031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494050176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.589039:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.589043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762476.589046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050176:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762476.589052:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494050304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762476.589072:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762476.589084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762476/real 1789762476] req@ffff8eb64304d500 x1876699494050304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762476.589098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494050304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762476.589107:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762476.589110:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762476.589113:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494050304:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762477.292526:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762478.316532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762479.340606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762480.364532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762481.388543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762481.708503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.708518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762481.708523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.708536:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2486 00000100:00080000:0.0:1789762481.708550:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.708559:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762481.708613:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494050432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.708640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.708643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.708644:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.708648:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2486 00000100:00080000:0.0:1789762481.708651:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.708654:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762481.708663:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762481.708674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494050560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.708683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.708686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.708689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762481.708691:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494050432:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762481.708694:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2486 00000100:00080000:0.0:1789762481.708701:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.708704:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762481.708708:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762481.708717:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494050688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762481.708725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762481.708729:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762481.708730:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762481.708734:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2486 00000100:00080000:0.0:1789762481.708752:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762481.708756:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762481.708760:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762481.708767:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494050816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762481.708774:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762481.708824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff8eb64304ce00 x1876699494050432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.708841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494050432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.708851:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.708854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762481.708858:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494050432:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762481.708881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.708900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762481.708909:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff8eb64304dc00 x1876699494050560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.708918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494050560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.708929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.708931:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762481.708934:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050560:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762481.708937:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050688:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762481.708950:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762481.708956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff8eb64304c380 x1876699494050688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.708963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494050688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.708969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.708970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762481.708972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494050688:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762481.708976:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494050816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762481.708987:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762481.708993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762481/real 1789762481] req@ffff8eb64304f800 x1876699494050816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762481.709001:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494050816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762481.709006:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762481.709007:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762481.709010:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494050816:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762482.412508:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762483.436497:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762484.460539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762485.484585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762486.508541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762486.828507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762486.828515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762486.828518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762486.828526:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2491 00000100:00080000:3.0:1789762486.828536:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762486.828542:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762486.828583:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494050944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762486.828606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762486.828608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762486.828610:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762486.828612:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2491 00000100:00080000:3.0:1789762486.828615:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762486.828617:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762486.828625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762486.828633:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494051072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762486.828641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762486.828644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762486.828646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762486.828649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2491 00000100:00080000:3.0:1789762486.828651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762486.828654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762486.828657:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762486.828665:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494050944:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762486.828668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494051200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762486.828678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762486.828680:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762486.828681:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762486.828684:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2491 00000100:00080000:3.0:1789762486.828686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762486.828688:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762486.828691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762486.828702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494051328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762486.828724:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762486.828757:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff8eb645148700 x1876699494050944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762486.828772:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494050944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762486.828779:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762486.828782:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762486.828785:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494050944:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762486.828790:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762486.828803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762486.828810:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff8eb64514b480 x1876699494051072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762486.828817:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494051072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762486.828823:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762486.828825:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762486.828827:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762486.828844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051200:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762486.828855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762486.828861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff8eb645149180 x1876699494051200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762486.828868:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494051200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762486.828874:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762486.828875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762486.828877:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762486.828880:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494051328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762486.828889:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762486.828895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762486/real 1789762486] req@ffff8eb64514ad80 x1876699494051328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762486.828903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494051328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762486.828907:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762486.828909:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762486.828911:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494051328:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762487.532572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762488.556633:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762489.580628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762490.604496:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762491.628537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762491.948494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762491.948503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762491.948507:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762491.948515:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2496 00000100:00080000:1.0:1789762491.948523:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762491.948529:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762491.948560:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494051456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762491.948588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762491.948591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762491.948592:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762491.948595:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2496 00000100:00080000:1.0:1789762491.948598:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762491.948601:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762491.948610:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762491.948619:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494051584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762491.948626:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494051456:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762491.948629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762491.948634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762491.948636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762491.948638:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2496 00000100:00080000:1.0:1789762491.948640:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762491.948643:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762491.948646:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762491.948654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494051712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762491.948661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762491.948663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762491.948664:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762491.948668:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2496 00000100:00080000:1.0:1789762491.948674:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762491.948677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762491.948680:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762491.948688:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762491.948689:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494051840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762491.948726:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762491/real 1789762491] req@ffff8eb65016ed80 x1876699494051456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762491.948740:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494051456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762491.948748:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762491.948751:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762491.948754:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494051456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762491.948772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762491.948788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762491.948797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762491/real 1789762491] req@ffff8eb65016f480 x1876699494051584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762491.948805:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494051584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762491.948810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762491.948813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762491.948815:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051584:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762491.948818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051712:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762491.948829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762491.948834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762491/real 1789762491] req@ffff8eb65016dc00 x1876699494051712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762491.948846:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494051712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762491.948854:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762491.948857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762491.948861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494051712:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762491.948867:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494051840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762491.948889:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762491.948900:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762491/real 1789762491] req@ffff8eb65016df80 x1876699494051840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762491.948908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494051840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762491.948914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762491.948916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762491.948918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494051840:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762492.652502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762493.676502:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762494.700529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762495.724533:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762496.748517:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762497.068502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762497.068511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762497.068515:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762497.068524:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2501 00000100:00080000:1.0:1789762497.068534:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762497.068539:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762497.068572:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494051968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762497.068600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762497.068602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762497.068604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762497.068607:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2501 00000100:00080000:1.0:1789762497.068610:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762497.068612:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762497.068620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762497.068631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494052096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762497.068643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762497.068654:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494051968:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762497.068654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762497.068664:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762497.068672:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2501 00000100:00080000:1.0:1789762497.068684:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762497.068689:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762497.068694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762497.068708:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494052224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762497.068716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762497.068719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762497.068720:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762497.068725:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2501 00000100:00080000:1.0:1789762497.068734:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762497.068736:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762497.068740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762497.068747:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494052352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762497.068756:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762497.068809:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff8eb65016c700 x1876699494051968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.068826:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494051968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.068835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.068839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762497.068842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494051968:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762497.068868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762497.068887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762497.068896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff8eb65016ce00 x1876699494052096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.068904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494052096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.068915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.068917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762497.068920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052096:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762497.068924:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052224:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762497.068935:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762497.068941:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff8eb65016c380 x1876699494052224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.068948:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494052224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.068954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.068956:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762497.068957:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052224:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762497.068961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494052352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762497.068972:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762497.068977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762497/real 1789762497] req@ffff8eb65016fb80 x1876699494052352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762497.068985:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494052352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762497.068989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762497.068991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762497.068994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494052352:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762497.772543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762498.796539:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762499.820560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762500.844532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762501.868496:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762502.188494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.188505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762502.188514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.188525:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2506 00000100:00080000:2.0:1789762502.188536:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.188544:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762502.188578:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494052480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.188602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.188606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.188608:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.188612:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2506 00000100:00080000:2.0:1789762502.188617:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.188622:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762502.188632:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762502.188637:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494052480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762502.188648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494052608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.188660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.188664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.188666:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.188671:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2506 00000100:00080000:2.0:1789762502.188675:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762502.188679:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762502.188684:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762502.188692:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762502.188698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494052736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762502.188710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762502.188714:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762502.188716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762502.188721:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2506 00000100:00080000:2.0:1789762502.188725:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762502.188728:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff8eb650f82a00 x1876699494052480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762502.188741:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762502.188745:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494052480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762502.188749:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762502.188756:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.188761:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762502.188768:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494052480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762502.188769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494052864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762502.188791:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.188817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762502.188831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff8eb650f83480 x1876699494052608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.188845:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494052608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.188852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.188855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762502.188857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762502.188864:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052736:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762502.188880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762502.188890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff8eb650f80e00 x1876699494052736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.188902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494052736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.188911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.188915:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762502.188919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494052736:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762502.188925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494052864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762502.188945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762502.188953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762502/real 1789762502] req@ffff8eb650f80700 x1876699494052864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762502.188961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494052864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762502.188966:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762502.188968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762502.188970:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494052864:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762502.892532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762503.916509:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762504.940534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762505.964516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762506.988537:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762507.308481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762507.308489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762507.308493:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762507.308501:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2511 00000100:00080000:3.0:1789762507.308510:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762507.308516:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762507.308550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494052992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762507.308573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762507.308575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762507.308577:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762507.308580:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2511 00000100:00080000:3.0:1789762507.308582:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762507.308585:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762507.308592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762507.308602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494053120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762507.308613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762507.308617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762507.308619:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762507.308623:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2511 00000100:00080000:3.0:1789762507.308627:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762507.308630:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494052992:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762507.308634:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762507.308643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762507.308667:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494053248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762507.308676:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762507.308679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762507.308680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762507.308683:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2511 00000100:00080000:3.0:1789762507.308687:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762507.308689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762507.308694:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762507.308703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494053376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762507.308711:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762507.308754:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff8eb64514ad80 x1876699494052992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.308772:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494052992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.308793:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.308797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762507.308801:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494052992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762507.308808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.308831:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762507.308844:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff8eb645149180 x1876699494053120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.308856:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494053120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.308864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.308867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762507.308870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053120:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762507.308893:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053248:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762507.308913:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762507.308923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff8eb64514b480 x1876699494053248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.308941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494053248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.308949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.308953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762507.308956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053248:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762507.308961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494053376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762507.308979:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762507.308991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762507/real 1789762507] req@ffff8eb645148700 x1876699494053376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762507.309003:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494053376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762507.309011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762507.309014:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762507.309018:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494053376:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762508.012507:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762509.036572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762510.060524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762511.084495:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762512.108501:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762512.428517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.428525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762512.428528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.428536:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2516 00000100:00080000:0.0:1789762512.428545:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.428551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762512.428598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494053504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.428628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.428631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762512.428633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.428636:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2516 00000100:00080000:0.0:1789762512.428639:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.428643:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762512.428651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.428662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494053632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762512.428671:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494053504:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762512.428672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.428681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762512.428684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.428688:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2516 00000100:00080000:0.0:1789762512.428692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.428696:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762512.428702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762512.428716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494053760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762512.428727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762512.428730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762512.428734:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762512.428736:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762512.428742:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2516 00000100:00080000:0.0:1789762512.428746:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762512.428749:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762512.428754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762512.428763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494053888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762512.428771:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff8eb64304e300 x1876699494053504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762512.428788:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494053504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762512.428800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762512.428804:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762512.428807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494053504:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762512.428832:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762512.428855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762512.428868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff8eb64304c000 x1876699494053632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762512.428878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494053632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762512.428884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762512.428886:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762512.428888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762512.428893:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053760:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762512.428907:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762512.428914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff8eb64304d180 x1876699494053760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762512.428921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494053760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762512.428927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762512.428929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762512.428931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494053760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762512.428935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494053888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762512.428946:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762512.428952:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762512/real 1789762512] req@ffff8eb64304ed80 x1876699494053888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762512.428959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494053888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762512.428964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762512.428966:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762512.428968:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494053888:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762513.132538:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762514.156601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762515.180599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762516.204602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762517.228544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762517.548490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762517.548498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762517.548501:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762517.548509:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2522 00000100:00080000:3.0:1789762517.548518:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762517.548524:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762517.548553:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494054016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762517.548574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762517.548577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762517.548578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762517.548581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2522 00000100:00080000:3.0:1789762517.548583:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762517.548586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762517.548593:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762517.548601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494054144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762517.548608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762517.548610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762517.548611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762517.548613:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2522 00000100:00080000:3.0:1789762517.548615:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762517.548616:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762517.548620:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762517.548628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494054272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762517.548635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762517.548637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762517.548639:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762517.548642:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2522 00000100:00080000:3.0:1789762517.548644:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762517.548648:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762517.548651:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494054016:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762517.548652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762517.548663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494054400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762517.548724:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762517.548756:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff8eb645148000 x1876699494054016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762517.548770:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494054016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762517.548778:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762517.548781:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762517.548784:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494054016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762517.548789:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762517.548802:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762517.548809:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff8eb645149880 x1876699494054144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762517.548818:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494054144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762517.548824:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762517.548826:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762517.548828:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054144:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762517.548831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054272:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762517.548841:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762517.548847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff8eb64514b100 x1876699494054272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762517.548854:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494054272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762517.548859:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762517.548861:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762517.548863:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054272:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762517.548881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494054400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762517.548891:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762517.548897:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762517/real 1789762517] req@ffff8eb64514bb80 x1876699494054400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762517.548904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494054400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762517.548914:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762517.548916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762517.548918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494054400:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762518.252540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762519.276513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762520.300535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762521.324525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762522.348616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762522.668487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762522.668497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762522.668502:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762522.668513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2527 00000100:00080000:3.0:1789762522.668525:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762522.668532:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762522.668575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494054528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762522.668610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762522.668613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762522.668615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762522.668618:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2527 00000100:00080000:3.0:1789762522.668621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762522.668624:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762522.668633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762522.668643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494054656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762522.668651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762522.668654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762522.668655:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762522.668658:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2527 00000100:00080000:3.0:1789762522.668660:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762522.668663:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762522.668667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762522.668672:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494054528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762522.668679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494054784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762522.668693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762522.668696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762522.668697:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762522.668700:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2527 00000100:00080000:3.0:1789762522.668703:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762522.668705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762522.668708:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762522.668716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494054912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762522.668752:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762522.668804:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff8eb64514a300 x1876699494054528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762522.668822:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494054528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762522.668831:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762522.668834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762522.668837:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494054528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762522.668843:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762522.668862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762522.668870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff8eb645148e00 x1876699494054656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762522.668878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494054656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762522.668885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762522.668887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762522.668889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762522.668915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054784:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762522.668927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762522.668934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff8eb645148380 x1876699494054784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762522.668941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494054784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762522.668947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762522.668949:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762522.668950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494054784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762522.668954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494054912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762522.668965:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762522.668970:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762522/real 1789762522] req@ffff8eb64514b800 x1876699494054912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762522.668978:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494054912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762522.668983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762522.668984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762522.668986:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494054912:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762523.372568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762524.396501:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762525.420503:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762526.444555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762527.468519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762527.788487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762527.788495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762527.788499:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762527.788507:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2532 00000100:00080000:2.0:1789762527.788516:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762527.788522:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762527.788555:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494055040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762527.788578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762527.788582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762527.788583:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762527.788586:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2532 00000100:00080000:2.0:1789762527.788588:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762527.788591:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762527.788599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762527.788607:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494055168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762527.788615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762527.788618:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762527.788619:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762527.788622:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2532 00000100:00080000:2.0:1789762527.788624:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762527.788626:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762527.788630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762527.788640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494055296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762527.788643:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494055040:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762527.788652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762527.788655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762527.788656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762527.788658:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2532 00000100:00080000:2.0:1789762527.788660:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762527.788663:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762527.788666:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762527.788675:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494055424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762527.788717:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762527.788766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff8eb650f82680 x1876699494055040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762527.788788:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494055040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762527.788796:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762527.788799:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762527.788802:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494055040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762527.788808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762527.788826:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762527.788835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff8eb650f82d80 x1876699494055168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762527.788843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494055168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762527.788848:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762527.788850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762527.788852:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762527.788871:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055296:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762527.788883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762527.788889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff8eb650f81880 x1876699494055296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762527.788896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494055296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762527.788906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762527.788908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762527.788911:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055296:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762527.788914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494055424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762527.788923:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762527.788929:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762527/real 1789762527] req@ffff8eb650f81c00 x1876699494055424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762527.788936:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494055424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762527.788941:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762527.788943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762527.788945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494055424:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762528.492591:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762529.516505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762530.540554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762531.564504:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762532.588517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762532.908492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762532.908503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762532.908507:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762532.908519:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2537 00000100:00080000:2.0:1789762532.908531:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762532.908539:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762532.908577:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494055552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762532.908602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762532.908605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762532.908606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762532.908609:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2537 00000100:00080000:2.0:1789762532.908613:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762532.908616:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762532.908624:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762532.908632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494055680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762532.908639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762532.908642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762532.908644:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762532.908646:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2537 00000100:00080000:2.0:1789762532.908649:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762532.908652:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762532.908663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762532.908672:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494055552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762532.908673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494055808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762532.908687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762532.908690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762532.908691:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762532.908693:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2537 00000100:00080000:2.0:1789762532.908695:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762532.908698:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762532.908700:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762532.908708:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494055936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762532.908742:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762532.908781:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762532/real 1789762532] req@ffff8eb650f81c00 x1876699494055552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762532.908800:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494055552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762532.908812:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762532.908816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762532.908821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494055552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762532.908829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762532.908849:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762532.908860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762532/real 1789762532] req@ffff8eb650f81880 x1876699494055680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762532.908872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494055680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762532.908881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762532.908885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762532.908889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055680:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762532.908912:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055808:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762532.908944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762532.908956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762532/real 1789762532] req@ffff8eb650f82d80 x1876699494055808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762532.908968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494055808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762532.908977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762532.908981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762532.908984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494055808:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762532.908991:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494055936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762532.909010:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762532.909021:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762532/real 1789762532] req@ffff8eb650f82680 x1876699494055936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762532.909035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494055936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762532.909042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762532.909046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762532.909049:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494055936:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762533.612634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762534.636521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762535.660542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762536.684546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762537.708524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762538.028481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762538.028489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762538.028492:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762538.028500:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2542 00000100:00080000:1.0:1789762538.028510:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762538.028517:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762538.028556:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494056064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762538.028577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762538.028580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762538.028581:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762538.028587:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2542 00000100:00080000:1.0:1789762538.028589:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762538.028592:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762538.028600:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762538.028608:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494056192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762538.028620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762538.028623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762538.028624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762538.028627:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2542 00000100:00080000:1.0:1789762538.028629:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762538.028632:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762538.028637:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494056064:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762538.028637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762538.028648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494056320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762538.028655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762538.028658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762538.028659:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762538.028662:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2542 00000100:00080000:1.0:1789762538.028664:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762538.028667:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762538.028670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762538.028678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494056448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762538.028718:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762538.028748:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762538.028764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494056448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762538.732537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762539.756520:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762540.780535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762541.100758:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762541.100803:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762541.100820:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762541.100853:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762541.100864:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762541.100912:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762541.100913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762541] req@ffff8eb65016d500 x1876699494056064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762541.100941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494056064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762541.100954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762541.100959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762541.100963:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494056064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762541.100971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762541] req@ffff8eb65016e300 x1876699494056192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762541.100984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494056192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762541.100993:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762541.100996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762541.101001:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762541.101023:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762541] req@ffff8eb65016d880 x1876699494056320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762541.101042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494056320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762541.101057:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762541.101061:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762541.101064:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762541.101072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762538/real 1789762541] req@ffff8eb65016ea00 x1876699494056448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762541.101084:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494056448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762541.101093:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762541.101097:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762541.101100:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494056448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762541.804541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762542.828555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762543.148528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762543.148544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762543.148549:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762543.148561:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2547 00000100:00080000:2.0:1789762543.148575:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762543.148583:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762543.148629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494056576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762543.148664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762543.148672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762543.148674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762543.148678:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2547 00000100:00080000:2.0:1789762543.148683:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762543.148689:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762543.148702:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762543.148716:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494056704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762543.148729:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762543.148733:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762543.148737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762543.148744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494056576:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762543.148753:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2547 00000100:00080000:2.0:1789762543.148761:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762543.148766:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762543.148773:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762543.148793:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494056832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762543.148807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762543.148811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762543.148813:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762543.148818:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2547 00000100:00080000:2.0:1789762543.148823:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762543.148829:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762543.148841:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789762543.148841:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762543.148879:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494056960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762543.148921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff8eb650f80700 x1876699494056576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762543.148946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494056576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762543.148959:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762543.148964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762543.148969:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494056576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762543.148997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762543.149027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762543.149042:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff8eb650f80e00 x1876699494056704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762543.149058:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494056704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762543.149068:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762543.149071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762543.149075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056704:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762543.149083:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056832:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762543.149107:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762543.149121:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff8eb650f83480 x1876699494056832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762543.149135:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494056832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762543.149144:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762543.149147:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762543.149150:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494056832:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762543.149158:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494056960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762543.149180:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762543.149193:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762543/real 1789762543] req@ffff8eb650f82a00 x1876699494056960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762543.149213:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494056960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762543.149222:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762543.149225:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762543.149229:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494056960:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762543.852513:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762544.876561:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762545.900556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762546.924580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762547.948600:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762548.268532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762548.268545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762548.268550:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762548.268564:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2552 00000100:00080000:0.0:1789762548.268577:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762548.268585:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762548.268632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494057088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762548.268671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762548.268675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762548.268678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762548.268683:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2552 00000100:00080000:0.0:1789762548.268687:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762548.268691:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762548.268703:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762548.268721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494057216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762548.268736:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762548.268741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762548.268747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762548.268748:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494057088:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762548.268755:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2552 00000100:00080000:0.0:1789762548.268766:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762548.268771:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762548.268778:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762548.268798:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494057344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762548.268811:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762548.268842:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762548.268847:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762548.268855:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2552 00000100:00080000:0.0:1789762548.268863:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762548.268871:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762548.268882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789762548.268889:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762548.268921:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494057472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762548.268937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff8eb64304ed80 x1876699494057088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762548.269002:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494057088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762548.269025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762548.269034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762548.269043:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494057088:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762548.269078:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762548.269116:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762548.269141:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff8eb64304d180 x1876699494057216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762548.269167:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494057216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762548.269186:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762548.269193:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762548.269201:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057216:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762548.269214:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057344:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762548.269276:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762548.269309:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff8eb64304c000 x1876699494057344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762548.269337:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494057344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762548.269354:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762548.269361:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762548.269368:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057344:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762548.269380:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494057472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762548.269457:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762548.269481:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762548/real 1789762548] req@ffff8eb64304e300 x1876699494057472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762548.269508:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494057472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762548.269524:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762548.269531:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762548.269538:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494057472:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762548.972521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762549.996546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762551.020544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762552.044528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762553.068553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762553.388468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762553.388475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762553.388478:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762553.388486:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2557 00000100:00080000:3.0:1789762553.388496:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762553.388502:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762553.388532:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494057600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762553.388554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762553.388556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762553.388561:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762553.388564:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2557 00000100:00080000:3.0:1789762553.388566:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762553.388569:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762553.388582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762553.388591:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494057728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762553.388600:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494057600:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762553.388600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762553.388609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762553.388611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762553.388614:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2557 00000100:00080000:3.0:1789762553.388616:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762553.388618:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762553.388622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762553.388630:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494057856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762553.388637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762553.388639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762553.388641:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762553.388644:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2557 00000100:00080000:3.0:1789762553.388647:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762553.388650:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762553.388653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762553.388660:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494057984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762553.388664:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762553.388699:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff8eb645149500 x1876699494057600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762553.388715:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494057600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762553.388723:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762553.388727:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762553.388730:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494057600:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762553.388746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762553.388760:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762553.388768:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff8eb645148a80 x1876699494057728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762553.388791:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494057728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762553.388800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762553.388804:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762553.388808:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762553.388814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057856:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762553.388843:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762553.388851:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff8eb64514a680 x1876699494057856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762553.388859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494057856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762553.388865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762553.388867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762553.388869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494057856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762553.388873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494057984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762553.388885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762553.388891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762553/real 1789762553] req@ffff8eb645149c00 x1876699494057984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762553.388898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494057984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762553.388903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762553.388904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762553.388906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494057984:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762554.092513:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762555.116519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762556.140531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762557.164557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762558.188576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762558.508478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762558.508488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762558.508492:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762558.508504:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2563 00000100:00080000:0.0:1789762558.508516:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762558.508530:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762558.508576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494058112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762558.508601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762558.508604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762558.508605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762558.508608:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2563 00000100:00080000:0.0:1789762558.508611:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762558.508613:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762558.508622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762558.508632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494058240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762558.508640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762558.508643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762558.508644:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762558.508647:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2563 00000100:00080000:0.0:1789762558.508649:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762558.508653:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762558.508657:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762558.508665:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494058112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762558.508666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494058368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762558.508681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762558.508684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762558.508685:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762558.508688:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2563 00000100:00080000:0.0:1789762558.508689:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762558.508692:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762558.508695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762558.508702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494058496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762558.508759:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762558.508802:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762558] req@ffff8eb64304f800 x1876699494058112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762558.508818:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494058112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762558.508828:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762558.508831:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762558.508834:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494058112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762558.508840:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762558.508857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762558.508865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762558] req@ffff8eb64304c380 x1876699494058240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762558.508874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494058240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762558.508880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762558.508882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762558.508884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058240:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762558.508907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058368:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762558.508920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762558.508926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762558] req@ffff8eb64304dc00 x1876699494058368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762558.508934:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494058368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762558.508939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762558.508941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762558.508943:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058368:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762558.508947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494058496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762558.508957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762558.508963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762558/real 1789762558] req@ffff8eb64304ce00 x1876699494058496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762558.508970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494058496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762558.508980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762558.508982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762558.508985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494058496:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762559.212594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762560.236541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762561.260508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762562.284522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762563.308520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762563.628499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762563.628507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762563.628510:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762563.628519:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2568 00000100:00080000:1.0:1789762563.628528:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762563.628533:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762563.628563:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494058624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762563.628586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762563.628589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762563.628590:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762563.628593:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2568 00000100:00080000:1.0:1789762563.628596:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762563.628599:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762563.628606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762563.628615:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494058752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762563.628621:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494058624:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762563.628628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762563.628644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762563.628646:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762563.628649:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2568 00000100:00080000:1.0:1789762563.628652:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762563.628654:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762563.628658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762563.628668:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494058880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762563.628676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762563.628678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762563.628679:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762563.628682:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2568 00000100:00080000:1.0:1789762563.628686:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762563.628688:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762563.628691:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762563.628696:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762563.628712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494059008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762563.628756:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff8eb65016f800 x1876699494058624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762563.628770:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494058624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762563.628779:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762563.628783:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762563.628786:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494058624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762563.628806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762563.628822:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762563.628830:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff8eb65016e680 x1876699494058752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762563.628838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494058752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762563.628843:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762563.628846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762563.628848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762563.628851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058880:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762563.628862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762563.628869:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff8eb65016ea00 x1876699494058880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762563.628876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494058880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762563.628881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762563.628882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762563.628884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494058880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762563.628887:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494059008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762563.628897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762563.628903:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762563/real 1789762563] req@ffff8eb65016d880 x1876699494059008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762563.628910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494059008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762563.628915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762563.628917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762563.628918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494059008:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762564.332515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762565.356518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762566.380528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762567.404565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762568.428534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762568.748500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762568.748512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762568.748517:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762568.748529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2573 00000100:00080000:1.0:1789762568.748541:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762568.748548:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762568.748590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494059136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762568.748624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762568.748628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762568.748630:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762568.748636:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2573 00000100:00080000:1.0:1789762568.748640:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762568.748646:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762568.748679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762568.748690:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494059136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762568.748699:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494059264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762568.748710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762568.748713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762568.748714:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762568.748718:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2573 00000100:00080000:1.0:1789762568.748721:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762568.748725:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762568.748729:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762568.748746:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494059392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762568.748756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762568.748759:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762568.748761:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762568.748764:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2573 00000100:00080000:1.0:1789762568.748769:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762568.748774:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762568.748777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762568.748783:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762568.748795:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494059520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762568.748832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff8eb65016e300 x1876699494059136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762568.748849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494059136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762568.748857:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762568.748860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762568.748864:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494059136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762568.748884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762568.748902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762568.748910:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff8eb65016d500 x1876699494059264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762568.748919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494059264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762568.748924:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762568.748926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762568.748928:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762568.748932:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059392:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762568.748943:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762568.748949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff8eb65016fb80 x1876699494059392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762568.748956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494059392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762568.748961:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762568.748963:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762568.748965:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059392:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762568.748968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494059520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762568.748978:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762568.748984:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762568/real 1789762568] req@ffff8eb65016c380 x1876699494059520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762568.748996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494059520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762568.749001:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762568.749002:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762568.749004:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494059520:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762569.452548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762570.476567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762571.500517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762572.524537:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762573.548547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762573.868491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762573.868499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762573.868513:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762573.868521:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2578 00000100:00080000:3.0:1789762573.868531:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762573.868537:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762573.868576:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494059648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762573.868604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762573.868607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762573.868608:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762573.868612:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2578 00000100:00080000:3.0:1789762573.868615:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762573.868618:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762573.868638:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762573.868650:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494059776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762573.868657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762573.868660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762573.868661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762573.868665:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2578 00000100:00080000:3.0:1789762573.868678:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762573.868681:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762573.868685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762573.868691:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494059904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762573.868699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494059648:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762573.868702:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762573.868709:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762573.868710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762573.868713:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2578 00000100:00080000:3.0:1789762573.868715:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762573.868717:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762573.868720:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762573.868729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494060032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762573.868781:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762573.868825:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762573/real 1789762573] req@ffff8eb645149c00 x1876699494059648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762573.868838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494059648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762573.868847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762573.868850:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762573.868853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494059648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762573.868858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762573.868872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762573.868880:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762573/real 1789762573] req@ffff8eb64514a680 x1876699494059776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762573.868892:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494059776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762573.868897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762573.868899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762573.868901:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059776:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762573.868920:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059904:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762573.868931:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762573.868937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762573/real 1789762573] req@ffff8eb645148a80 x1876699494059904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762573.868945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494059904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762573.868950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762573.868951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762573.868953:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494059904:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762573.868957:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494060032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762573.868966:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762573.868972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762573/real 1789762573] req@ffff8eb645149500 x1876699494060032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762573.868979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494060032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762573.868988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762573.868990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762573.868992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494060032:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762574.572532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762575.596521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762576.620512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762577.644554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762578.668699:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762578.988528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762578.988541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762578.988546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762578.988560:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2583 00000100:00080000:0.0:1789762578.988575:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762578.988603:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762578.988658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494060160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762578.988692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762578.988696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762578.988699:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762578.988703:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2583 00000100:00080000:0.0:1789762578.988707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762578.988711:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762578.988723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762578.988735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494060288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762578.988752:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762578.988757:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762578.988759:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762578.988763:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2583 00000100:00080000:0.0:1789762578.988766:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762578.988770:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762578.988775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762578.988787:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494060416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762578.988800:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762578.988804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494060160:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762578.988805:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762578.988810:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762578.988823:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2583 00000100:00080000:0.0:1789762578.988827:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762578.988830:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762578.988835:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762578.988849:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494060544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762578.988909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762578.988971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762578/real 1789762578] req@ffff8eb64304d500 x1876699494060160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762578.988996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494060160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762578.989008:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762578.989012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762578.989017:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494060160:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762578.989024:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762578.989046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762578.989058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762578/real 1789762578] req@ffff8eb64304d880 x1876699494060288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762578.989069:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494060288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762578.989077:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762578.989080:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762578.989084:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762578.989113:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060416:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762578.989131:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762578.989140:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762578/real 1789762578] req@ffff8eb64304f480 x1876699494060416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762578.989151:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494060416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762578.989159:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762578.989162:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762578.989165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762578.989170:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494060544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762578.989184:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762578.989193:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762578/real 1789762578] req@ffff8eb64304e680 x1876699494060544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762578.989203:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494060544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762578.989211:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762578.989213:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762578.989216:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494060544:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762579.692621:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762580.716694:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762581.740577:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762582.764513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762583.788564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762584.108516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762584.108525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762584.108528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762584.108537:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2588 00000100:00080000:0.0:1789762584.108557:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762584.108564:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762584.108597:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494060672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762584.108617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762584.108619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762584.108620:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762584.108623:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2588 00000100:00080000:0.0:1789762584.108626:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762584.108628:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762584.108636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762584.108645:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494060800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762584.108653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762584.108656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762584.108656:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494060672:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762584.108660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762584.108669:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2588 00000100:00080000:0.0:1789762584.108671:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762584.108673:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762584.108676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762584.108684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494060928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762584.108692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762584.108694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762584.108696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762584.108698:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2588 00000100:00080000:0.0:1789762584.108712:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762584.108715:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762584.108718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762584.108726:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494061056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762584.108730:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762584.108774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff8eb64304ea00 x1876699494060672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762584.108795:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494060672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762584.108807:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762584.108812:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762584.108818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494060672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762584.108840:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762584.108864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762584.108887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff8eb64304c700 x1876699494060800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762584.108902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494060800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762584.108911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762584.108914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762584.108918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060800:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762584.108925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060928:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762584.108945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762584.108956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff8eb64304f100 x1876699494060928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762584.108969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494060928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762584.108984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762584.108988:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762584.108991:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494060928:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762584.108997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494061056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762584.109014:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762584.109025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762584/real 1789762584] req@ffff8eb64304ca80 x1876699494061056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762584.109045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494061056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762584.109053:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762584.109056:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762584.109060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494061056:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762584.812521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762585.836526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762586.860492:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762587.884534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762588.908633:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762589.228528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762589.228538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762589.228543:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762589.228554:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2593 00000100:00080000:0.0:1789762589.228565:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762589.228573:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762589.228620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494061184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762589.228647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762589.228650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762589.228652:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762589.228656:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2593 00000100:00080000:0.0:1789762589.228659:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762589.228662:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762589.228672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762589.228681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494061312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762589.228690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762589.228693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762589.228694:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762589.228696:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2593 00000100:00080000:0.0:1789762589.228699:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762589.228702:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494061184:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762589.228704:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762589.228712:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762589.228719:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494061440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762589.228726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762589.228729:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762589.228730:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762589.228732:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2593 00000100:00080000:0.0:1789762589.228735:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762589.228737:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762589.228740:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762589.228747:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494061568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762589.228777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762589.228825:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff8eb64304ca80 x1876699494061184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762589.228844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494061184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762589.228853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762589.228857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762589.228861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494061184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762589.228884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762589.228901:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762589.228910:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff8eb64304f100 x1876699494061312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762589.228919:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494061312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762589.228925:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762589.228927:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762589.228930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061312:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762589.228936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061440:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762589.228949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762589.228955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff8eb64304c700 x1876699494061440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762589.228962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494061440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762589.228967:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762589.228969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762589.228971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762589.228975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494061568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762589.228985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762589.228991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762589/real 1789762589] req@ffff8eb64304ea00 x1876699494061568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762589.229000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494061568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762589.229005:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762589.229006:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762589.229008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494061568:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762589.932549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762590.956555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762591.980526:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762593.004578:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762594.028571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762594.348508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762594.348516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762594.348519:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762594.348527:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2598 00000100:00080000:1.0:1789762594.348536:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762594.348542:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762594.348574:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494061696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762594.348597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762594.348599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762594.348601:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762594.348604:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2598 00000100:00080000:1.0:1789762594.348606:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762594.348609:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762594.348618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762594.348627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494061824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762594.348632:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494061696:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762594.348648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762594.348650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762594.348651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762594.348655:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2598 00000100:00080000:1.0:1789762594.348657:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762594.348659:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762594.348663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762594.348672:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494061952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762594.348678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762594.348681:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762594.348683:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762594.348686:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2598 00000100:00080000:1.0:1789762594.348688:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762594.348691:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762594.348696:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762594.348698:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762594.348712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494062080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762594.348743:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff8eb65016ce00 x1876699494061696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762594.348763:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494061696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762594.348771:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762594.348775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762594.348777:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494061696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762594.348797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762594.348824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762594.348835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff8eb65016c700 x1876699494061824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762594.348843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494061824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762594.348849:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762594.348851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762594.348853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762594.348859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061952:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762594.348872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762594.348878:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff8eb65016df80 x1876699494061952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762594.348886:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494061952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762594.348891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762594.348893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762594.348895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494061952:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762594.348898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494062080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762594.348909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762594.348915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762594/real 1789762594] req@ffff8eb65016dc00 x1876699494062080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762594.348922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494062080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762594.348927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762594.348928:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762594.348930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494062080:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762595.052562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762596.076532:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762597.100543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762598.124541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762599.148584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762599.468493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762599.468502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762599.468505:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762599.468513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2603 00000100:00080000:1.0:1789762599.468523:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762599.468529:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762599.468562:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494062208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762599.468584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762599.468587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762599.468588:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762599.468591:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2603 00000100:00080000:1.0:1789762599.468593:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762599.468595:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762599.468603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762599.468611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494062336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762599.468619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762599.468621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762599.468623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762599.468628:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2603 00000100:00100000:0.0:1789762599.468631:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494062208:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762599.468631:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762599.468638:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762599.468643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762599.468650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494062464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762599.468658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762599.468661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762599.468662:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762599.468665:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2603 00000100:00080000:1.0:1789762599.468667:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762599.468670:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762599.468673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762599.468679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494062592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762599.468696:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762599.468730:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff8eb65016f480 x1876699494062208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762599.468752:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494062208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762599.468760:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762599.468763:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762599.468766:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494062208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762599.468771:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762599.468786:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762599.468800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff8eb65016ed80 x1876699494062336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762599.468808:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494062336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762599.468814:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762599.468816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762599.468818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062336:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762599.468836:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062464:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762599.468849:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762599.468856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff8eb65016d180 x1876699494062464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762599.468868:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494062464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762599.468873:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762599.468875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762599.468877:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062464:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762599.468880:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494062592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762599.468890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762599.468896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762599/real 1789762599] req@ffff8eb65016c000 x1876699494062592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762599.468909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494062592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762599.468915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762599.468917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762599.468919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494062592:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762600.172577:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762601.196562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762602.220563:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762603.244531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762604.268510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762604.588498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762604.588506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762604.588512:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762604.588523:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2609 00000100:00080000:3.0:1789762604.588534:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762604.588542:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762604.588578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494062720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762604.588602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762604.588606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762604.588608:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762604.588613:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2609 00000100:00080000:3.0:1789762604.588618:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762604.588622:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762604.588632:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494062720:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762604.588632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762604.588653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494062848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762604.588672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762604.588676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762604.588678:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762604.588682:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2609 00000100:00080000:3.0:1789762604.588686:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762604.588690:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762604.588696:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762604.588711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494062976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762604.588721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762604.588725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762604.588728:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762604.588730:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062848:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789762604.588736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2609 00000100:00080000:3.0:1789762604.588742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762604.588746:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762604.588753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762604.588766:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789762604.588780:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494063104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762604.588802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494063104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762605.292545:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762606.316541:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762607.340500:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762607.660638:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789762607.660674:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762607.660688:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762607.660717:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762607.660723:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762607.660728:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762607.660766:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762607] req@ffff8eb64514a300 x1876699494063104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762607.660792:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494063104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762607.660811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762607.660816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762607.660820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494063104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762607.660829:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762607] req@ffff8eb645148380 x1876699494062848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762607.660843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494062848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762607.660853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762607.660857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762607.660861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062848:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762607.660868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762607] req@ffff8eb645148e00 x1876699494062976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762607.660882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494062976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762607.660898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762607.660901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762607.660904:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494062976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762607.660911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762604/real 1789762607] req@ffff8eb64514b800 x1876699494062720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762607.660923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494062720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762607.660931:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762607.660934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762607.660937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494062720:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762608.364576:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762609.388505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762609.708523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762609.708534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762609.708539:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762609.708550:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2614 00000100:00080000:3.0:1789762609.708562:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762609.708569:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762609.708611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494063232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762609.708638:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762609.708641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762609.708643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762609.708651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2614 00000100:00080000:3.0:1789762609.708654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762609.708657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762609.708667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762609.708680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494063360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762609.708693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762609.708698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762609.708707:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494063232:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762609.708710:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762609.708718:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2614 00000100:00080000:3.0:1789762609.708725:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762609.708729:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762609.708735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762609.708750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494063488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762609.708768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762609.708777:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762609.708780:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762609.708784:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2614 00000100:00080000:3.0:1789762609.708797:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762609.708803:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762609.708811:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762609.708812:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762609.708829:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494063616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762609.708878:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff8eb64514bb80 x1876699494063232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.708902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494063232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.708915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.708919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762609.708924:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494063232:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762609.708957:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762609.708985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762609.709000:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff8eb64514b100 x1876699494063360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.709014:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494063360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.709026:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.709030:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762609.709034:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762609.709040:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063488:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762609.709065:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762609.709077:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff8eb645149880 x1876699494063488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.709089:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494063488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.709099:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.709102:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762609.709106:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063488:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762609.709113:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494063616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762609.709136:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762609.709149:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762609/real 1789762609] req@ffff8eb645148000 x1876699494063616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762609.709161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494063616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762609.709169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762609.709171:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762609.709176:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494063616:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762610.412513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762611.436495:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762612.460542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762613.484458:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762614.508484:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762614.828467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762614.828479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762614.828483:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762614.828494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2619 00000100:00080000:3.0:1789762614.828506:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762614.828513:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762614.828570:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494063744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762614.828599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762614.828602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762614.828603:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762614.828607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2619 00000100:00080000:3.0:1789762614.828610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762614.828614:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762614.828624:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762614.828636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494063872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762614.828645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762614.828649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762614.828652:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494063744:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762614.828652:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762614.828659:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2619 00000100:00080000:3.0:1789762614.828663:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762614.828666:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762614.828669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762614.828677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494064000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762614.828686:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762614.828688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762614.828690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762614.828693:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2619 00000100:00080000:3.0:1789762614.828695:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762614.828697:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762614.828700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762614.828707:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494064128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762614.828731:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762614.828777:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff8eb645148700 x1876699494063744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.828799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494063744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.828812:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.828817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762614.828822:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494063744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762614.828846:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.828864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762614.828874:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff8eb64514b480 x1876699494063872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.828882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494063872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.828888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.828891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762614.828893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494063872:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762614.828897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064000:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762614.828908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762614.828915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff8eb645149180 x1876699494064000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.828923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494064000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.828928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.828929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762614.828931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064000:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762614.828935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494064128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762614.828945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762614.828951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762614/real 1789762614] req@ffff8eb64514ad80 x1876699494064128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762614.828959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494064128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762614.828964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762614.828966:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762614.828968:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494064128:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762615.532587:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762616.556516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762617.580497:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762618.604504:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762619.628502:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762619.948469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762619.948483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762619.948488:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762619.948498:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2624 00000100:00080000:3.0:1789762619.948510:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762619.948519:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762619.948556:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494064256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762619.948579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762619.948584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762619.948586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762619.948591:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2624 00000100:00080000:3.0:1789762619.948596:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762619.948601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762619.948611:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494064256:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762619.948612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762619.948632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494064384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762619.948642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762619.948645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762619.948646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762619.948649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2624 00000100:00080000:3.0:1789762619.948653:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762619.948655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762619.948660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762619.948669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494064512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762619.948675:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762619.948683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762619.948686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762619.948688:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762619.948692:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2624 00000100:00080000:3.0:1789762619.948697:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762619.948699:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762619.948702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762619.948712:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494064640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762619.948717:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762619/real 1789762619] req@ffff8eb64514ad80 x1876699494064256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762619.948735:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494064256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762619.948744:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762619.948747:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762619.948750:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494064256:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762619.948769:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762619.948783:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762619.948791:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762619/real 1789762619] req@ffff8eb645149180 x1876699494064384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762619.948799:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494064384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762619.948805:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762619.948807:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762619.948809:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762619.948812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064512:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762619.948823:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762619.948829:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762619/real 1789762619] req@ffff8eb64514b480 x1876699494064512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762619.948841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494064512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762619.948850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762619.948853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762619.948857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762619.948863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494064640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762619.948884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762619.948896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762619/real 1789762619] req@ffff8eb645148700 x1876699494064640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762619.948905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494064640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762619.948910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762619.948912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762619.948914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494064640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762620.652536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762621.676528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762622.700578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762623.724580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762624.748503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762625.068515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762625.068526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762625.068531:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762625.068550:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2629 00000100:00080000:3.0:1789762625.068562:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762625.068569:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762625.068607:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494064768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762625.068635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762625.068639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762625.068642:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762625.068647:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2629 00000100:00080000:3.0:1789762625.068652:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762625.068657:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762625.068667:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762625.068677:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494064768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762625.068684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494064896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762625.068699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762625.068703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762625.068705:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762625.068709:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2629 00000100:00080000:3.0:1789762625.068714:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762625.068719:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762625.068733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762625.068737:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762625.068754:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494065024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762625.068766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762625.068770:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762625.068773:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762625.068775:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762625] req@ffff8eb645148000 x1876699494064768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762625.068783:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2629 00000100:00080000:3.0:1789762625.068791:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762625.068797:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494064768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762625.068798:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762625.068809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762625.068811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762625.068818:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762625.068821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494064768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762625.068827:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494065152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762625.068844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762625.068863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762625.068873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762625] req@ffff8eb645149880 x1876699494064896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762625.068882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494064896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762625.068888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762625.068891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762625.068895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494064896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762625.068899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065024:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762625.068910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762625.068916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762625] req@ffff8eb64514b100 x1876699494065024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762625.068923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494065024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762625.068928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762625.068930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762625.068932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065024:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762625.068935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494065152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762625.068944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762625.068950:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762625/real 1789762625] req@ffff8eb64514bb80 x1876699494065152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762625.068957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494065152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762625.068966:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762625.068968:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762625.068970:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494065152:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762625.772495:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762626.796505:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762627.820530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762628.844503:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762629.868539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762630.188525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762630.188539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762630.188544:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762630.188559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2634 00000100:00080000:0.0:1789762630.188573:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762630.188583:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762630.188632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494065280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762630.188664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762630.188668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762630.188670:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762630.188675:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2634 00000100:00080000:0.0:1789762630.188679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762630.188683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762630.188695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762630.188711:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494065408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762630.188724:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762630.188735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762630.188735:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494065280:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762630.188740:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762630.188752:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2634 00000100:00080000:0.0:1789762630.188757:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762630.188763:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762630.188771:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762630.188789:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494065536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762630.188801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762630.188806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762630.188808:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762630.188816:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2634 00000800:00000100:1.0:1789762630.188817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762630.188827:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762630.188835:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762630.188843:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762630.188860:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494065664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762630.188873:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff8eb64304e680 x1876699494065280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.188900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494065280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.188913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.188919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762630.188926:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494065280:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762630.188953:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762630.188980:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762630.188992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff8eb64304f480 x1876699494065408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.189005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494065408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.189014:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.189017:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762630.189021:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762630.189029:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065536:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762630.189052:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762630.189065:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff8eb64304d880 x1876699494065536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.189077:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494065536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.189086:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.189089:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762630.189093:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762630.189100:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494065664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762630.189119:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762630.189128:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762630/real 1789762630] req@ffff8eb64304d500 x1876699494065664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762630.189140:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494065664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762630.189148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762630.189152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762630.189156:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494065664:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762630.892540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762631.916536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762632.940542:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762633.964578:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762634.988561:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762635.308539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.308560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762635.308566:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.308580:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2639 00000100:00080000:0.0:1789762635.308594:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.308602:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762635.308653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494065792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.308679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.308682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.308684:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.308687:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2639 00000100:00080000:0.0:1789762635.308690:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.308693:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762635.308704:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762635.308715:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494065920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.308722:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.308725:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.308726:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.308729:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2639 00000100:00080000:0.0:1789762635.308732:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.308734:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762635.308738:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762635.308745:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494065792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762635.308752:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494066048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762635.308762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762635.308765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762635.308767:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762635.308773:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2639 00000100:00080000:0.0:1789762635.308776:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762635.308779:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762635.308782:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762635.308792:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494066176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762635.308818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762635.308867:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff8eb64304ce00 x1876699494065792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.308886:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494065792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.308895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.308899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762635.308902:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494065792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762635.308907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.308922:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762635.308929:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff8eb64304dc00 x1876699494065920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.308937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494065920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.308943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.308945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762635.308947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494065920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762635.308968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066048:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762635.308979:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762635.308985:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff8eb64304c380 x1876699494066048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.308993:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494066048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.309002:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.309004:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762635.309006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066048:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762635.309010:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494066176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762635.309020:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762635.309025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762635/real 1789762635] req@ffff8eb64304f800 x1876699494066176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762635.309032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494066176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762635.309037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762635.309039:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762635.309041:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494066176:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762636.012629:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762637.036556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762638.060493:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762639.084503:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762640.108558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762640.428459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.428467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762640.428470:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.428478:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2644 00000100:00080000:0.0:1789762640.428487:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.428492:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762640.428525:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494066304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.428546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.428548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762640.428549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.428551:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2644 00000100:00080000:0.0:1789762640.428553:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.428556:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762640.428564:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.428571:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494066432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.428578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.428580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762640.428582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.428584:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2644 00000100:00080000:0.0:1789762640.428586:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.428588:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762640.428591:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762640.428596:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494066560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762640.428603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762640.428605:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762640.428607:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762640.428611:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2644 00000100:00080000:0.0:1789762640.428613:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762640.428615:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762640.428618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762640.428623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494066688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762640.428634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494066304:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789762640.428697:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762640.428729:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff8eb64304e300 x1876699494066304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762640.428744:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494066304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762640.428752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762640.428755:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762640.428758:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494066304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762640.428763:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762640.428789:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762640.428797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff8eb64304c000 x1876699494066432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762640.428805:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494066432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762640.428810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762640.428813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762640.428815:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762640.428818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066560:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762640.428829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762640.428840:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff8eb64304d180 x1876699494066560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762640.428850:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494066560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762640.428855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762640.428857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762640.428859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066560:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762640.428879:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494066688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762640.428895:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762640.428901:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762640/real 1789762640] req@ffff8eb64304ed80 x1876699494066688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762640.428909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494066688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762640.428915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762640.428916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762640.428918:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494066688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762641.132518:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762642.156552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762643.180522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762644.204508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762645.228566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762645.548507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762645.548517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762645.548522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762645.548532:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2650 00000100:00080000:3.0:1789762645.548542:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762645.548548:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762645.548585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494066816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762645.548611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762645.548614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762645.548615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762645.548619:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2650 00000100:00080000:3.0:1789762645.548622:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762645.548625:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762645.548633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762645.548644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494066944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762645.548657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762645.548661:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494066816:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762645.548661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762645.548665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762645.548673:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2650 00000100:00080000:3.0:1789762645.548675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762645.548677:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762645.548680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762645.548689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494067072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762645.548702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762645.548705:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762645.548706:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762645.548710:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2650 00000100:00080000:3.0:1789762645.548715:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762645.548718:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762645.548721:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762645.548736:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762645.548742:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494067200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762645.548771:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff8eb64514b800 x1876699494066816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762645.548786:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494066816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762645.548794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762645.548797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762645.548800:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494066816:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762645.548817:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762645.548831:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762645.548838:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff8eb645148e00 x1876699494066944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762645.548846:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494066944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762645.548856:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762645.548859:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762645.548861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494066944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762645.548864:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067072:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762645.548874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762645.548880:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff8eb645148380 x1876699494067072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762645.548887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494067072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762645.548892:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762645.548894:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762645.548896:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762645.548899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494067200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762645.548909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762645.548914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762645/real 1789762645] req@ffff8eb64514a300 x1876699494067200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762645.548921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494067200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762645.548926:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762645.548928:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762645.548930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494067200:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762646.252500:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762647.276567:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762648.300523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762649.324591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762650.348547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762650.668560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762650.668572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762650.668578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762650.668591:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2655 00000100:00080000:3.0:1789762650.668604:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762650.668613:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762650.668663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494067328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762650.668690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762650.668693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762650.668695:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762650.668699:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2655 00000100:00080000:3.0:1789762650.668702:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762650.668705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762650.668716:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762650.668727:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494067456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762650.668737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762650.668740:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762650.668742:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762650.668745:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2655 00000100:00080000:3.0:1789762650.668748:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762650.668754:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762650.668757:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494067328:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762650.668759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762650.668773:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494067584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762650.668790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762650.668793:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762650.668795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762650.668798:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2655 00000100:00080000:3.0:1789762650.668802:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762650.668805:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762650.668808:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762650.668818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494067712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762650.668848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762650.668914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff8eb645149500 x1876699494067328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762650.668934:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494067328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762650.668945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762650.668950:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762650.668954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494067328:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762650.668960:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762650.668979:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762650.668989:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff8eb645148a80 x1876699494067456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762650.669000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494067456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762650.669007:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762650.669010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762650.669013:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067456:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762650.669039:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067584:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762650.669083:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762650.669115:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff8eb64514a680 x1876699494067584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762650.669138:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494067584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762650.669149:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762650.669153:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762650.669158:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067584:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762650.669166:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494067712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762650.669193:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762650.669208:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762650/real 1789762650] req@ffff8eb645149c00 x1876699494067712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762650.669219:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494067712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762650.669226:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762650.669228:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762650.669231:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494067712:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762651.372528:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762652.396503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762653.420532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762654.444552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762655.468592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762655.788551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762655.788566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762655.788572:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762655.788587:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2660 00000100:00080000:3.0:1789762655.788603:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762655.788614:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762655.788677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494067840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762655.788719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762655.788724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762655.788727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762655.788733:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2660 00000100:00080000:3.0:1789762655.788740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762655.788758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762655.788772:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762655.788784:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494067840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762655.788802:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494067968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762655.788823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762655.788828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762655.788831:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762655.788837:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2660 00000100:00080000:3.0:1789762655.788845:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762655.788857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762655.788857:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762655.788869:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762655.788888:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494068096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762655.788900:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff8eb645149c00 x1876699494067840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762655.788907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762655.788920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762655.788924:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762655.788929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494067840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762655.788938:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2660 00000100:00080000:3.0:1789762655.788948:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762655.788957:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:1.0:1789762655.788966:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762655.788970:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762655.788972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762655.788984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494067840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762655.788998:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494068224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762655.789023:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762655.789057:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762655.789080:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff8eb64514a680 x1876699494067968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762655.789089:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494067968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762655.789097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762655.789100:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762655.789103:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494067968:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762655.789110:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068096:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762655.789123:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762655.789129:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff8eb645148a80 x1876699494068096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762655.789137:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494068096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762655.789143:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762655.789145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762655.789147:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762655.789158:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494068224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762655.789169:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762655.789175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762655/real 1789762655] req@ffff8eb645149500 x1876699494068224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762655.789183:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494068224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762655.789187:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762655.789189:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762655.789191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494068224:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762656.492512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762657.516549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762658.540564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762659.564591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762660.588520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762660.908458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762660.908467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762660.908471:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762660.908480:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2665 00000100:00080000:3.0:1789762660.908489:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762660.908495:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762660.908528:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494068352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762660.908549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762660.908552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762660.908553:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762660.908556:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2665 00000100:00080000:3.0:1789762660.908558:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762660.908561:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762660.908568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762660.908576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494068480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762660.908583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762660.908585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762660.908586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762660.908588:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2665 00000100:00080000:3.0:1789762660.908590:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762660.908592:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762660.908595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762660.908602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494068608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762660.908609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762660.908612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762660.908613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762660.908616:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2665 00000100:00080000:3.0:1789762660.908618:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762660.908620:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762660.908623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762660.908629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494068736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762660.908639:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494068352:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789762660.908695:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762660.908732:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762660/real 1789762660] req@ffff8eb64514a300 x1876699494068352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762660.908747:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494068352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762660.908754:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762660.908757:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762660.908760:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494068352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762660.908765:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762660.908779:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762660.908786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762660/real 1789762660] req@ffff8eb645148380 x1876699494068480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762660.908794:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494068480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762660.908799:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762660.908801:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762660.908803:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762660.908807:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068608:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762660.908816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762660.908822:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762660/real 1789762660] req@ffff8eb645148e00 x1876699494068608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762660.908829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494068608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762660.908834:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762660.908836:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762660.908837:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068608:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762660.908856:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494068736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762660.908866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762660.908872:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762660/real 1789762660] req@ffff8eb64514b800 x1876699494068736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762660.908879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494068736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762660.908888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762660.908891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762660.908893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494068736:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762661.612509:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762662.636545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762663.660511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762664.684560:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762665.708532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762666.028508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762666.028519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762666.028523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762666.028535:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2670 00000100:00080000:3.0:1789762666.028547:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762666.028555:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762666.028599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494068864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762666.028629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762666.028633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762666.028635:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762666.028638:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2670 00000100:00080000:3.0:1789762666.028640:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762666.028643:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762666.028653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762666.028662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494068992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762666.028669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762666.028672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762666.028673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762666.028676:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2670 00000100:00080000:3.0:1789762666.028678:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762666.028680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762666.028683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762666.028695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494069120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762666.028704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762666.028708:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789762666.028709:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494068864:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762666.028710:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762666.028716:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2670 00000100:00080000:3.0:1789762666.028720:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762666.028723:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762666.028727:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762666.028735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494069248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762666.028797:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762666.028846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff8eb64514bb80 x1876699494068864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.028866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494068864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.028878:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.028882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762666.028886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494068864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762666.028894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.028917:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762666.028929:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff8eb64514b100 x1876699494068992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.028959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494068992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.028968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.028972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762666.028976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494068992:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762666.029003:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069120:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762666.029025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762666.029035:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff8eb645149880 x1876699494069120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.029047:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494069120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.029055:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.029058:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762666.029062:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069120:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762666.029068:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494069248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762666.029086:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762666.029097:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762666/real 1789762666] req@ffff8eb645148000 x1876699494069248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762666.029109:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494069248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762666.029118:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762666.029121:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762666.029125:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494069248:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762666.732565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762667.756549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762668.780517:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762669.804495:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762670.828542:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762671.148522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762671.148536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762671.148541:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762671.148555:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2675 00000100:00080000:3.0:1789762671.148568:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762671.148575:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762671.148620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494069376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762671.148648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762671.148652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762671.148654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762671.148659:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2675 00000100:00080000:3.0:1789762671.148663:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762671.148667:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762671.148678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762671.148692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494069504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762671.148702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762671.148706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762671.148709:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762671.148710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494069376:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762671.148713:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2675 00000100:00080000:3.0:1789762671.148716:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762671.148720:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762671.148725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762671.148734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494069632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762671.148744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762671.148747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762671.148749:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762671.148752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2675 00000100:00080000:3.0:1789762671.148755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762671.148758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762671.148763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762671.148771:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494069760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762671.148841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762671.148875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762671.148892:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494069760:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762671.852508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762672.876554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762673.900555:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762674.220786:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762674.220840:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762674.220857:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762674.220918:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762674.220928:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762674.220938:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762674.220981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762674] req@ffff8eb64514b480 x1876699494069504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762674.221028:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494069504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762674.221048:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762674.221054:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762674.221060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762674.221071:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762674] req@ffff8eb645149180 x1876699494069632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762674.221087:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494069632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762674.221097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762674.221101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762674.221105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494069632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762674.221116:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762674] req@ffff8eb64514ad80 x1876699494069760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762674.221129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494069760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762674.221138:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762674.221142:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762674.221146:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494069760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762674.221155:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762671/real 1789762674] req@ffff8eb645148700 x1876699494069376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762674.221169:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494069376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762674.221178:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762674.221182:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762674.221186:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494069376:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762674.924644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762675.948527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762676.268461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762676.268472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762676.268476:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762676.268486:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2680 00000100:00080000:3.0:1789762676.268497:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762676.268505:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762676.268538:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494069888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762676.268562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762676.268565:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762676.268566:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762676.268570:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2680 00000100:00080000:3.0:1789762676.268574:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762676.268578:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762676.268594:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494069888:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762676.268597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762676.268614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494070016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762676.268627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762676.268631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762676.268633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762676.268636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2680 00000100:00080000:3.0:1789762676.268641:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762676.268646:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762676.268653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762676.268659:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762676.268668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494070144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762676.268681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762676.268685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762676.268687:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762676.268692:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff8eb645148700 x1876699494069888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762676.268693:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2680 00000100:00080000:3.0:1789762676.268702:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762676.268709:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762676.268715:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494069888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762676.268717:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762676.268736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494070272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762676.268762:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.268770:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762676.268775:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494069888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762676.268802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762676.268829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762676.268841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff8eb64514ad80 x1876699494070016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.268852:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494070016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.268884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.268889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762676.268892:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762676.268899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070144:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762676.268922:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762676.268934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff8eb645149180 x1876699494070144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.268947:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494070144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.268956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.268959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762676.268962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070144:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762676.268969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494070272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762676.268998:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762676.269010:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762676/real 1789762676] req@ffff8eb64514b480 x1876699494070272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762676.269022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494070272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762676.269029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762676.269033:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762676.269036:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494070272:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762676.972563:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762677.996535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762679.020564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762680.044521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762681.068569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762681.388496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762681.388507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762681.388511:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762681.388522:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2685 00000100:00080000:3.0:1789762681.388533:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762681.388541:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762681.388584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494070400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762681.388613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762681.388616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762681.388617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762681.388621:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2685 00000100:00080000:3.0:1789762681.388623:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762681.388626:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762681.388635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762681.388644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494070528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762681.388655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762681.388658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762681.388659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762681.388662:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2685 00000100:00080000:3.0:1789762681.388664:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762681.388667:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762681.388672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762681.388678:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494070656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762681.388687:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762681.388689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762681.388691:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762681.388698:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2685 00000100:00080000:3.0:1789762681.388701:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762681.388702:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494070400:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762681.388718:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762681.388722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762681.388735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494070784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762681.388790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762681.388845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff8eb645148000 x1876699494070400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.388862:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494070400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.388874:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.388878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762681.388883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494070400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762681.388891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.388916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762681.388926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff8eb645149880 x1876699494070528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.388934:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494070528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.388940:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.388942:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762681.388945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070528:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762681.388965:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070656:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762681.388978:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762681.388984:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff8eb64514b100 x1876699494070656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.388991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494070656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.388996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.388998:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762681.389000:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494070656:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762681.389004:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494070784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762681.389014:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762681.389020:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762681/real 1789762681] req@ffff8eb64514bb80 x1876699494070784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762681.389032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494070784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762681.389037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762681.389038:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762681.389040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494070784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762682.092548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762683.116505:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762684.140500:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762685.164696:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762686.188647:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762686.508518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.508528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762686.508532:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.508543:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2691 00000100:00080000:1.0:1789762686.508555:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.508562:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762686.508605:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494070912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.508629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.508632:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.508633:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.508637:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2691 00000100:00080000:1.0:1789762686.508639:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.508641:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762686.508650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762686.508661:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494071040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.508671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.508673:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.508675:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.508678:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2691 00000100:00080000:1.0:1789762686.508685:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762686.508688:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494070912:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762686.508689:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762686.508698:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762686.508713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494071168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762686.508723:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762686.508726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762686.508727:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762686.508731:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2691 00000100:00080000:1.0:1789762686.508735:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762686.508738:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762686.508741:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762686.508748:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494071296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762686.508763:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762686.508820:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff8eb65016f100 x1876699494070912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.508834:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494070912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.508842:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.508846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762686.508848:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494070912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762686.508870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.508885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762686.508893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff8eb65016ca80 x1876699494071040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.508906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494071040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.508911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.508913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762686.508915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762686.508919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071168:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762686.508930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762686.508936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff8eb65016c000 x1876699494071168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.508943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494071168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.508948:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.508950:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762686.508952:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762686.508955:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494071296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762686.508965:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762686.508971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762686/real 1789762686] req@ffff8eb65016d180 x1876699494071296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762686.508977:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494071296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762686.508986:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762686.508988:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762686.508990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494071296:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762687.212631:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762688.236595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762689.260548:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762690.284614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762691.308531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762691.628525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762691.628536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762691.628541:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762691.628554:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2696 00000100:00080000:1.0:1789762691.628566:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762691.628574:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762691.628622:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494071424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762691.628652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762691.628655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762691.628656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762691.628659:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2696 00000100:00080000:1.0:1789762691.628662:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762691.628664:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762691.628675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762691.628685:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494071552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762691.628694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762691.628698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762691.628702:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762691.628704:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494071424:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762691.628707:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2696 00000100:00080000:1.0:1789762691.628714:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762691.628722:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762691.628728:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762691.628761:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494071680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762691.628782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762691.628786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762691.628789:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762691.628794:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2696 00000100:00080000:1.0:1789762691.628800:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762691.628806:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762691.628809:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762691.628815:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762691.628828:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494071808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762691.628858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762691] req@ffff8eb65016ed80 x1876699494071424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762691.628885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494071424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762691.628894:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762691.628898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762691.628901:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494071424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762691.628923:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762691.628942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762691.628950:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762691] req@ffff8eb65016f480 x1876699494071552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762691.628959:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494071552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762691.628969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762691.628971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762691.628973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071552:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762691.628977:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071680:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762691.628989:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762691.628995:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762691] req@ffff8eb65016dc00 x1876699494071680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762691.629003:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494071680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762691.629008:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762691.629010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762691.629012:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494071680:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762691.629016:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494071808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762691.629027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762691.629039:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762691/real 1789762691] req@ffff8eb65016df80 x1876699494071808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762691.629047:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494071808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762691.629052:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762691.629055:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762691.629057:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494071808:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762692.332507:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762693.356494:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762694.380562:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762695.404498:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762696.428574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762696.748522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762696.748530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762696.748533:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762696.748541:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2701 00000100:00080000:0.0:1789762696.748551:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762696.748557:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762696.748588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494071936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762696.748611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762696.748614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762696.748615:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762696.748618:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2701 00000100:00080000:0.0:1789762696.748620:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762696.748623:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762696.748631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762696.748639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494072064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762696.748648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762696.748649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494071936:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762696.748655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762696.748658:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762696.748661:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2701 00000100:00080000:0.0:1789762696.748664:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762696.748666:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762696.748670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762696.748678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494072192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762696.748685:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762696.748688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762696.748689:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762696.748692:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2701 00000100:00080000:0.0:1789762696.748697:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762696.748700:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762696.748714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762696.748724:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494072320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762696.748727:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762696.748774:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff8eb64304ed80 x1876699494071936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762696.748796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494071936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762696.748808:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762696.748813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762696.748818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494071936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762696.748837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762696.748855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762696.748864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff8eb64304d180 x1876699494072064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762696.748873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494072064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762696.748879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762696.748881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762696.748883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072064:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762696.748887:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072192:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762696.748899:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762696.748905:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff8eb64304c000 x1876699494072192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762696.748912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494072192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762696.748917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762696.748919:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762696.748921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762696.748925:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494072320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762696.748934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762696.748940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762696/real 1789762696] req@ffff8eb64304e300 x1876699494072320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762696.748947:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494072320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762696.748956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762696.748958:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762696.748961:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494072320:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762697.452577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762698.476530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762699.500504:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762700.524522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762701.548516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762701.868477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762701.868486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762701.868490:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762701.868498:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2706 00000100:00080000:1.0:1789762701.868507:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762701.868514:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762701.868545:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494072448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762701.868569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762701.868572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762701.868573:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762701.868576:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2706 00000100:00080000:1.0:1789762701.868578:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762701.868581:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762701.868589:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762701.868598:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494072576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762701.868603:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494072448:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762701.868608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762701.868612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762701.868614:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762701.868618:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2706 00000100:00080000:1.0:1789762701.868620:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762701.868622:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762701.868625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762701.868633:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494072704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762701.868640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762701.868642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762701.868644:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762701.868647:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2706 00000100:00080000:1.0:1789762701.868656:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762701.868659:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762701.868662:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762701.868682:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762701.868683:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494072832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762701.868725:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762701/real 1789762701] req@ffff8eb65016c700 x1876699494072448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762701.868743:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494072448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762701.868755:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762701.868759:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762701.868763:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494072448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762701.868784:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762701.868806:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762701.868818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762701/real 1789762701] req@ffff8eb65016ce00 x1876699494072576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762701.868830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494072576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762701.868839:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762701.868842:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762701.868845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762701.868852:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072704:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762701.868869:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762701.868878:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762701/real 1789762701] req@ffff8eb65016c380 x1876699494072704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762701.868890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494072704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762701.868898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762701.868902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762701.868905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494072704:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762701.868911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494072832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762701.868927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762701.868937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762701/real 1789762701] req@ffff8eb65016fb80 x1876699494072832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762701.868949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494072832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762701.868956:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762701.868959:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762701.868962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494072832:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762702.572549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762703.596516:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762704.620550:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762705.644490:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762706.668533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762706.988504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762706.988511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762706.988515:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762706.988523:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2711 00000100:00080000:3.0:1789762706.988532:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762706.988538:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762706.988569:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494072960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762706.988595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762706.988598:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762706.988599:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762706.988603:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2711 00000100:00080000:3.0:1789762706.988605:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762706.988607:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762706.988615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762706.988624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494073088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762706.988646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762706.988649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762706.988651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762706.988654:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2711 00000100:00080000:3.0:1789762706.988657:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762706.988663:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762706.988664:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494072960:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789762706.988668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762706.988681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494073216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762706.988692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762706.988697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762706.988699:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762706.988703:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2711 00000100:00080000:3.0:1789762706.988708:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762706.988713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762706.988718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762706.988730:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762706.988732:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494073344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762706.988782:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762706/real 1789762706] req@ffff8eb64514b800 x1876699494072960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762706.988807:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494072960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762706.988817:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762706.988822:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762706.988826:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494072960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762706.988837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762706.988859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762706.988871:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762706/real 1789762706] req@ffff8eb645148e00 x1876699494073088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762706.988885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494073088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762706.988894:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762706.988897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762706.988901:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073088:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762706.988924:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073216:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762706.988943:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762706.988953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762706/real 1789762706] req@ffff8eb645148380 x1876699494073216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762706.988964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494073216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762706.988973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762706.988977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762706.988980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073216:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762706.988987:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494073344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762706.989003:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762706.989013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762706/real 1789762706] req@ffff8eb64514a300 x1876699494073344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762706.989026:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494073344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762706.989033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762706.989036:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762706.989039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494073344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762707.692536:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762708.716515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762709.740525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762710.764502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762711.788535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762712.108498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.108506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762712.108510:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.108519:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2716 00000100:00080000:1.0:1789762712.108528:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.108536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762712.108575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494073472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.108633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.108636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.108637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.108641:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2716 00000100:00080000:1.0:1789762712.108645:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.108648:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762712.108656:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.108667:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494073600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762712.108675:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.108678:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.108679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.108682:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2716 00000100:00080000:1.0:1789762712.108684:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.108686:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762712.108689:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762712.108699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494073728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762712.108700:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494073472:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762712.108714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762712.108717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762712.108718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762712.108720:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2716 00000100:00080000:1.0:1789762712.108724:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762712.108726:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762712.108729:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762712.108737:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494073856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762712.108760:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762712.108793:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff8eb65016d500 x1876699494073472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762712.108806:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494073472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762712.108814:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762712.108817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762712.108819:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494073472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762712.108824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762712.108836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762712.108844:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff8eb65016e300 x1876699494073600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762712.108851:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494073600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762712.108857:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762712.108859:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762712.108862:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762712.108880:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073728:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762712.108891:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762712.108896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff8eb65016d880 x1876699494073728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762712.108903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494073728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762712.108909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762712.108910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762712.108912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494073728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762712.108916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494073856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762712.108925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762712.108930:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762712/real 1789762712] req@ffff8eb65016ea00 x1876699494073856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762712.108938:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494073856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762712.108942:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762712.108944:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762712.108947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494073856:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762712.812511:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762713.836506:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762714.860498:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762715.884505:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762716.908545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762717.228474:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762717.228483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762717.228486:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762717.228494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2721 00000100:00080000:1.0:1789762717.228503:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762717.228510:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762717.228542:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494073984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762717.228572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762717.228575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762717.228577:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762717.228580:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2721 00000100:00080000:1.0:1789762717.228583:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762717.228586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762717.228594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762717.228606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494074112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762717.228616:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494073984:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762717.228620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762717.228625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762717.228627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762717.228632:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2721 00000100:00080000:1.0:1789762717.228637:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762717.228641:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762717.228649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762717.228662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494074240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762717.228670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762717.228675:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762717.228677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762717.228681:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762717.228686:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2721 00000100:00080000:1.0:1789762717.228692:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762717.228696:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762717.228702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762717.228711:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff8eb65016e680 x1876699494073984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762717.228718:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494074368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762717.228730:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494073984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762717.228739:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762717.228743:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762717.228746:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494073984:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762717.228764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762717.228778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762717.228786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff8eb65016f800 x1876699494074112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762717.228793:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494074112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762717.228798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762717.228800:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762717.228802:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074112:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762717.228806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074240:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762717.228816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762717.228823:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff8eb65016ea00 x1876699494074240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762717.228830:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494074240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762717.228835:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762717.228837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762717.228838:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074240:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762717.228842:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494074368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762717.228851:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762717.228857:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762717/real 1789762717] req@ffff8eb65016d880 x1876699494074368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762717.228864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494074368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762717.228869:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762717.228870:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762717.228872:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494074368:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762717.932510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762718.956530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762719.980554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762721.004546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762722.028623:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762722.348494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762722.348503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762722.348506:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762722.348528:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2726 00000100:00080000:1.0:1789762722.348538:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762722.348544:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762722.348589:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494074496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762722.348612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762722.348615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762722.348617:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762722.348620:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2726 00000100:00080000:1.0:1789762722.348622:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762722.348624:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762722.348633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762722.348641:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494074624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762722.348649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762722.348652:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762722.348653:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762722.348656:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2726 00000100:00080000:1.0:1789762722.348658:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762722.348660:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762722.348664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762722.348672:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494074752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762722.348681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762722.348685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762722.348685:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494074496:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762722.348693:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762722.348706:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2726 00000100:00080000:1.0:1789762722.348710:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762722.348712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762722.348718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762722.348740:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494074880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762722.348787:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762722.348831:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff8eb65016e300 x1876699494074496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762722.348849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494074496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762722.348861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762722.348865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762722.348868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494074496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762722.348876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762722.348902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762722.348919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff8eb65016d500 x1876699494074624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762722.348936:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494074624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762722.348945:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762722.348948:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762722.348951:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074624:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762722.348979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074752:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762722.348999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762722.349024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff8eb65016fb80 x1876699494074752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762722.349032:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494074752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762722.349038:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762722.349040:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762722.349042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494074752:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762722.349048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494074880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762722.349060:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762722.349067:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762722/real 1789762722] req@ffff8eb65016c380 x1876699494074880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762722.349075:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494074880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762722.349080:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762722.349082:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762722.349084:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494074880:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762723.052573:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762724.076521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762725.100534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762726.124593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762727.148550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762727.468485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762727.468495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762727.468499:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762727.468509:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2731 00000100:00080000:1.0:1789762727.468521:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762727.468540:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762727.468578:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494075008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762727.468603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762727.468606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762727.468607:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762727.468610:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2731 00000100:00080000:1.0:1789762727.468614:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762727.468616:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762727.468625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762727.468635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494075136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762727.468643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762727.468649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762727.468649:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494075008:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762727.468650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762727.468658:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2731 00000100:00080000:1.0:1789762727.468661:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762727.468663:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762727.468667:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762727.468676:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494075264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762727.468698:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762727.468702:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762727.468704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762727.468709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2731 00000100:00080000:1.0:1789762727.468713:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762727.468718:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762727.468723:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762727.468732:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762727.468738:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494075392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762727.468789:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff8eb65016ce00 x1876699494075008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762727.468810:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494075008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762727.468822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762727.468825:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762727.468828:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494075008:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762727.468847:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762727.468866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762727.468875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff8eb65016c700 x1876699494075136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762727.468884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494075136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762727.468890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762727.468892:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762727.468894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075136:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762727.468898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075264:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762727.468909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762727.468916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff8eb65016df80 x1876699494075264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762727.468923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494075264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762727.468928:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762727.468930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762727.468932:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075264:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762727.468935:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494075392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762727.468945:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762727.468951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762727/real 1789762727] req@ffff8eb65016dc00 x1876699494075392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762727.468958:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494075392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762727.468967:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762727.468969:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762727.468971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494075392:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762728.172572:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762729.196567:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762730.220533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762731.244574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762732.268675:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762732.588484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.588492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762732.588495:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.588503:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2737 00000100:00080000:1.0:1789762732.588513:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.588518:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762732.588557:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494075520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.588580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.588583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.588587:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.588591:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2737 00000100:00080000:1.0:1789762732.588593:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.588596:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762732.588603:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762732.588613:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494075648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.588620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.588623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.588624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.588626:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2737 00000100:00080000:1.0:1789762732.588628:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762732.588633:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494075520:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762732.588634:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762732.588641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762732.588648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494075776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762732.588655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762732.588658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762732.588660:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762732.588663:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2737 00000100:00080000:1.0:1789762732.588666:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762732.588669:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762732.588672:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762732.588679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494075904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762732.588694:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762732.588736:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff8eb65016f480 x1876699494075520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762732.588756:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494075520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762732.588766:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762732.588769:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762732.588772:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494075520:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762732.588777:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762732.588795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762732.588805:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff8eb65016ed80 x1876699494075648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762732.588812:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494075648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762732.588818:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762732.588820:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762732.588823:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075648:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789762732.588841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075776:192.168.203.160@tcp:8:kworker.0 00000800:00000100:2.0:1789762732.588852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762732.588859:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff8eb65016d180 x1876699494075776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762732.588866:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494075776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762732.588871:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762732.588873:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762732.588875:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494075776:192.168.203.160@tcp:8:kworker.0 00000100:00100000:2.0:1789762732.588878:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494075904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:2.0:1789762732.588888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762732.588893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762732/real 1789762732] req@ffff8eb65016c000 x1876699494075904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762732.588900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494075904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762732.588905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762732.588907:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:2.0:1789762732.588909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494075904:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762733.292553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762734.316555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762735.340650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762736.364560:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762737.388641:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762737.708518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762737.708529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762737.708534:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762737.708545:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2742 00000100:00080000:1.0:1789762737.708557:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762737.708566:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762737.708615:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494076032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762737.708645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762737.708654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762737.708656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762737.708660:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2742 00000100:00080000:1.0:1789762737.708663:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762737.708666:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762737.708677:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762737.708688:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494076160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762737.708699:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494076032:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762737.708700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762737.708709:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762737.708712:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762737.708716:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2742 00000100:00080000:1.0:1789762737.708718:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762737.708721:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762737.708725:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762737.708736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494076288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762737.708744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762737.708746:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762737.708748:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762737.708751:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2742 00000100:00080000:1.0:1789762737.708765:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762737.708768:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762737.708773:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762737.708783:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494076416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762737.708834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076160:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762737.708867:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076288:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762737.708886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494076416:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762738.412510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762739.436546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762740.460543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762740.780686:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789762740.780734:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762740.780750:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762740.780789:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762740.780796:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789762740.780802:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762740.780861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762740] req@ffff8eb65016f100 x1876699494076160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762740.780894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494076160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762740.780910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762740.780917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762740.780922:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076160:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762740.780931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762740] req@ffff8eb65016c000 x1876699494076288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762740.780945:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494076288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762740.780954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762740.780957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762740.780960:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076288:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762740.780967:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762740] req@ffff8eb65016d180 x1876699494076416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762740.780979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494076416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762740.780987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762740.780991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762740.780994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494076416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762740.781001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762737/real 1789762740] req@ffff8eb65016ca80 x1876699494076032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762740.781013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494076032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762740.781022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762740.781025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762740.781029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494076032:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789762741.484571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762742.508604:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762742.828625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762742.828638:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762742.828643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762742.828658:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2747 00000100:00080000:2.0:1789762742.828672:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762742.828682:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762742.828735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494076544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762742.828784:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762742.828791:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762742.828795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762742.828803:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2747 00000100:00080000:2.0:1789762742.828810:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762742.828818:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762742.828834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762742.828869:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494076672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762742.828878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762742.828882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762742.828884:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762742.828888:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2747 00000100:00080000:2.0:1789762742.828893:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762742.828900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494076544:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762742.828901:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762742.828911:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762742.828927:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494076800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762742.828939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762742.828943:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762742.828945:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762742.828949:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2747 00000100:00080000:2.0:1789762742.828954:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762742.828958:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762742.828964:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762742.828976:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494076928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762742.829006:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762742.829061:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff8eb650f83800 x1876699494076544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.829084:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494076544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.829096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.829101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762742.829105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494076544:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762742.829114:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.829143:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762742.829157:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff8eb650f81180 x1876699494076672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.829176:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494076672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.829185:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.829189:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762742.829193:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076672:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762742.829226:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076800:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762742.829249:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762742.829262:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff8eb650f83b80 x1876699494076800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.829276:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494076800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.829285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.829289:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762742.829292:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494076800:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762742.829299:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494076928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762742.829323:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762742.829335:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762742/real 1789762742] req@ffff8eb650f81f80 x1876699494076928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762742.829377:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494076928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762742.829386:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762742.829389:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762742.829419:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494076928:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762743.532554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762744.556636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762745.580587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762746.604556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762747.628570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762747.948511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762747.948521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762747.948525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762747.948537:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2752 00000100:00080000:2.0:1789762747.948549:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762747.948556:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762747.948602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494077056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762747.948636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762747.948643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762747.948646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762747.948653:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2752 00000100:00080000:2.0:1789762747.948657:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762747.948663:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762747.948676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762747.948681:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494077056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762747.948697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494077184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762747.948712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762747.948715:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762747.948717:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762747.948721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2752 00000100:00080000:2.0:1789762747.948724:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762747.948728:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762747.948733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762747.948745:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494077312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762747.948754:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762747.948769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762747.948773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762747.948774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762747.948779:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2752 00000100:00080000:2.0:1789762747.948782:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762747.948784:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762747.948790:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762747.948802:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494077440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762747.948811:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762747/real 1789762747] req@ffff8eb650f80a80 x1876699494077056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762747.948829:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494077056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762747.948838:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762747.948842:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762747.948845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494077056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762747.948864:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762747.948881:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762747.948889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762747/real 1789762747] req@ffff8eb650f80000 x1876699494077184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762747.948903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494077184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762747.948908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762747.948910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762747.948912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077184:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762747.948916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077312:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762747.948928:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762747.948934:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762747/real 1789762747] req@ffff8eb650f80380 x1876699494077312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762747.948942:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494077312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762747.948947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762747.948949:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762747.948951:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077312:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762747.948955:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494077440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762747.948965:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762747.948971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762747/real 1789762747] req@ffff8eb650f81500 x1876699494077440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762747.948978:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494077440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762747.948988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762747.948990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762747.948992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494077440:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762748.652548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762749.676523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762750.700543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762751.724529:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762752.748555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762753.068508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762753.068516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762753.068520:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762753.068529:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2757 00000100:00080000:2.0:1789762753.068539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762753.068544:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762753.068578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494077568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762753.068598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762753.068601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762753.068602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762753.068605:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2757 00000100:00080000:2.0:1789762753.068607:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762753.068610:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762753.068618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762753.068627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494077696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762753.068634:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494077568:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762753.068638:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762753.068644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762753.068645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762753.068648:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2757 00000100:00080000:2.0:1789762753.068651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762753.068654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762753.068657:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762753.068665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494077824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762753.068673:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762753.068675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762753.068676:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762753.068680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2757 00000100:00080000:2.0:1789762753.068682:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762753.068684:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762753.068687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762753.068702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494077952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762753.068714:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762753.068752:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff8eb650f81500 x1876699494077568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762753.068771:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494077568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762753.068782:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762753.068787:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762753.068791:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494077568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762753.068814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762753.068835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762753.068846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff8eb650f80380 x1876699494077696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762753.068858:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494077696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762753.068872:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762753.068875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762753.068879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077696:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762753.068885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077824:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762753.068902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762753.068912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff8eb650f80000 x1876699494077824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762753.068922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494077824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762753.068929:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762753.068932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762753.068935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494077824:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762753.068941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494077952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762753.068957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762753.068966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762753/real 1789762753] req@ffff8eb650f80a80 x1876699494077952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762753.068978:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494077952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762753.068985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762753.068989:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762753.068992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494077952:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762753.772556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762754.796527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762755.820531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762756.844557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762757.868522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762758.188561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762758.188576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762758.188582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762758.188594:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2762 00000100:00080000:1.0:1789762758.188608:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762758.188617:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762758.188666:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494078080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762758.188700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762758.188703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762758.188705:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762758.188710:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2762 00000100:00080000:1.0:1789762758.188714:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762758.188719:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762758.188731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762758.188746:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494078208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762758.188758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762758.188761:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494078080:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762758.188763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762758.188768:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762758.188775:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2762 00000100:00080000:1.0:1789762758.188779:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762758.188783:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762758.188790:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762758.188802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494078336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762758.188813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762758.188817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762758.188820:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762758.188824:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2762 00000100:00080000:1.0:1789762758.188829:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762758.188833:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762758.188839:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762758.188844:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762758.188853:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494078464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762758.188890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762758] req@ffff8eb65016ed80 x1876699494078080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762758.188906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494078080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762758.188915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762758.188918:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762758.188921:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494078080:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762758.188945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762758.188960:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762758.188968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762758] req@ffff8eb65016f480 x1876699494078208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762758.188977:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494078208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762758.188982:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762758.188984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762758.188986:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078208:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762758.188992:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078336:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762758.189003:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762758.189009:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762758] req@ffff8eb65016dc00 x1876699494078336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762758.189017:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494078336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762758.189022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762758.189024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762758.189025:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078336:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762758.189029:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494078464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762758.189038:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762758.189044:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762758/real 1789762758] req@ffff8eb65016df80 x1876699494078464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762758.189051:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494078464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762758.189056:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762758.189058:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762758.189059:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494078464:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762758.892566:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762759.916510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762760.940502:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762761.964531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762762.988521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762763.308499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762763.308514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762763.308520:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762763.308534:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2767 00000100:00080000:1.0:1789762763.308549:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762763.308559:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762763.308610:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494078592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762763.308645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762763.308655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762763.308658:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762763.308664:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2767 00000100:00080000:1.0:1789762763.308668:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762763.308673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762763.308685:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494078592:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762763.308686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762763.308711:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494078720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762763.308727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762763.308732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762763.308734:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762763.308739:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2767 00000100:00080000:1.0:1789762763.308753:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762763.308760:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762763.308761:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762763.308775:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762763.308793:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494078848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762763.308807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762763.308808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff8eb65016c700 x1876699494078592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762763.308829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762763.308835:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762763.308843:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2767 00000100:00100000:0.0:1789762763.308844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494078592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762763.308852:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762763.308861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762763.308866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:1.0:1789762763.308878:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762763.308879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494078592:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762763.308897:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762763.308913:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762763.308921:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494078976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762763.308938:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762763.308953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff8eb65016ce00 x1876699494078720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762763.308968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494078720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762763.308977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762763.308982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762763.308985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762763.308993:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078848:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762763.309016:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762763.309027:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff8eb65016c380 x1876699494078848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762763.309040:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494078848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762763.309045:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762763.309048:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762763.309050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494078848:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762763.309062:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494078976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762763.309076:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762763.309082:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762763/real 1789762763] req@ffff8eb65016fb80 x1876699494078976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762763.309089:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494078976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762763.309094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762763.309096:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762763.309098:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494078976:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762764.012727:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762765.036655:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762766.060599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762767.084610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762768.108617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762768.428526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762768.428541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762768.428548:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762768.428565:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2772 00000100:00080000:1.0:1789762768.428586:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762768.428601:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762768.428673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494079104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762768.428738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762768.428745:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762768.428748:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762768.428755:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2772 00000100:00080000:1.0:1789762768.428762:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762768.428768:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762768.428784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762768.428808:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494079104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762768.428811:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494079232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762768.428839:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762768.428846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762768.428849:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762768.428857:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2772 00000100:00080000:1.0:1789762768.428862:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762768.428867:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762768.428875:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762768.428890:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494079360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762768.428902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762768.428921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762768.428926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762768.428929:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762768.428935:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2772 00000100:00080000:1.0:1789762768.428943:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762768.428948:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762768.428955:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762768.428960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff8eb65016d500 x1876699494079104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762768.428973:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494079488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762768.429011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494079104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762768.429024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762768.429028:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762768.429031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494079104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762768.429056:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762768.429075:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762768.429084:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff8eb65016e300 x1876699494079232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762768.429092:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494079232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762768.429098:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762768.429100:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762768.429102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762768.429106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079360:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762768.429117:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762768.429123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff8eb65016d880 x1876699494079360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762768.429130:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494079360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762768.429139:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762768.429141:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762768.429143:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079360:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762768.429147:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494079488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762768.429157:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762768.429163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762768/real 1789762768] req@ffff8eb65016ea00 x1876699494079488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762768.429192:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494079488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762768.429201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762768.429204:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762768.429208:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494079488:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762769.132655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762770.156570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762771.180591:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762772.204650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762773.228538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762773.548506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762773.548515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762773.548519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762773.548528:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2778 00000100:00080000:2.0:1789762773.548537:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762773.548543:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762773.548586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494079616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762773.548613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762773.548618:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762773.548621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762773.548630:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2778 00000100:00080000:2.0:1789762773.548633:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762773.548637:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762773.548647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762773.548652:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494079616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762773.548660:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494079744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762773.548670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762773.548673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762773.548675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762773.548677:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2778 00000100:00080000:2.0:1789762773.548680:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762773.548682:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762773.548685:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762773.548693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494079872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762773.548700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762773.548702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762773.548704:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762773.548706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2778 00000100:00080000:2.0:1789762773.548709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762773.548711:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762773.548717:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789762773.548717:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762773.548728:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494080000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762773.548783:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff8eb650f81f80 x1876699494079616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762773.548798:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494079616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762773.548806:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762773.548809:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762773.548812:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494079616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762773.548831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762773.548848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762773.548856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff8eb650f83b80 x1876699494079744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762773.548864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494079744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762773.548870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762773.548872:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762773.548875:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079744:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762773.548879:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079872:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762773.548890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762773.548896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff8eb650f81180 x1876699494079872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762773.548903:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494079872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762773.548908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762773.548910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762773.548912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494079872:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762773.548915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494080000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762773.548926:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762773.548931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762773/real 1789762773] req@ffff8eb650f83800 x1876699494080000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762773.548939:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494080000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762773.548943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762773.548945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762773.548947:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494080000:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762774.252595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762775.276523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762776.300538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762777.324647:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762778.348525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762778.668472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762778.668480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762778.668484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762778.668492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2783 00000100:00080000:0.0:1789762778.668502:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762778.668508:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762778.668538:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494080128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762778.668561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762778.668565:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762778.668567:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762778.668571:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2783 00000100:00080000:0.0:1789762778.668575:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762778.668580:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762778.668590:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762778.668601:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494080128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762778.668607:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494080256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762778.668618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762778.668621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762778.668622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762778.668625:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2783 00000100:00080000:0.0:1789762778.668628:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762778.668631:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762778.668635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762778.668644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494080384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762778.668652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762778.668654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762778.668656:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762778.668658:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2783 00000800:00000100:1.0:1789762778.668665:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789762778.668670:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762778.668675:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762778.668679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762778.668691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494080512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762778.668715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff8eb64304f800 x1876699494080128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762778.668732:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494080128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762778.668742:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762778.668745:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762778.668748:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494080128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762778.668775:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762778.668793:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762778.668802:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff8eb64304c380 x1876699494080256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762778.668810:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494080256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762778.668815:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762778.668817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762778.668820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762778.668823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080384:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762778.668833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762778.668839:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff8eb64304dc00 x1876699494080384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762778.668846:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494080384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762778.668855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762778.668857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762778.668859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080384:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762778.668863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494080512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762778.668872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762778.668877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762778/real 1789762778] req@ffff8eb64304ce00 x1876699494080512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762778.668884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494080512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762778.668889:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762778.668891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762778.668893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494080512:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762779.372512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762780.396536:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762781.420637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762782.444599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762783.468574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762783.788523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762783.788537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762783.788540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762783.788548:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2788 00000100:00080000:1.0:1789762783.788558:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762783.788563:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762783.788598:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494080640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762783.788620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762783.788623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762783.788625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762783.788628:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2788 00000100:00080000:1.0:1789762783.788631:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762783.788634:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762783.788642:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762783.788658:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494080768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762783.788668:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494080640:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762783.788672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762783.788683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762783.788685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762783.788689:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2788 00000100:00080000:1.0:1789762783.788692:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762783.788697:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762783.788703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762783.788720:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494080896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762783.788732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762783.788736:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762783.788739:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762783.788745:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2788 00000100:00080000:1.0:1789762783.788750:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762783.788756:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762783.788757:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762783.788770:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762783.788780:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494081024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762783.788821:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff8eb65016f800 x1876699494080640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762783.788842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494080640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762783.788854:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762783.788858:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762783.788861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494080640:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762783.788886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762783.788903:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762783.788912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff8eb65016e680 x1876699494080768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762783.788926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494080768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762783.788932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762783.788934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762783.788936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080768:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762783.788941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080896:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762783.788953:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762783.788959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff8eb65016ea00 x1876699494080896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762783.788967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494080896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762783.788972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762783.788974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762783.788976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494080896:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762783.788980:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494081024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762783.788991:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762783.788997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762783/real 1789762783] req@ffff8eb65016d880 x1876699494081024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762783.789005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494081024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762783.789015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762783.789017:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762783.789019:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494081024:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762784.492526:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762785.516510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762786.540551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762787.564575:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762788.588539:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762788.908489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762788.908497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762788.908500:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762788.908508:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2793 00000100:00080000:2.0:1789762788.908518:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762788.908523:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762788.908554:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494081152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762788.908573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762788.908576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762788.908577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762788.908581:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2793 00000100:00080000:2.0:1789762788.908583:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762788.908586:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762788.908593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762788.908602:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494081280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762788.908610:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494081152:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762788.908612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762788.908617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762788.908619:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762788.908622:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2793 00000100:00080000:2.0:1789762788.908625:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762788.908627:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762788.908631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762788.908645:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494081408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762788.908653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762788.908656:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762788.908657:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762788.908660:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2793 00000100:00080000:2.0:1789762788.908663:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762788.908665:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762788.908668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762788.908673:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762788.908679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494081536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762788.908726:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762788/real 1789762788] req@ffff8eb650f82a00 x1876699494081152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762788.908741:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494081152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762788.908752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762788.908756:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762788.908761:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494081152:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762788.908784:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762788.908808:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762788.908818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762788/real 1789762788] req@ffff8eb650f83480 x1876699494081280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762788.908827:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494081280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762788.908837:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762788.908839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762788.908842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081280:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762788.908845:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081408:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762788.908858:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762788.908864:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762788/real 1789762788] req@ffff8eb650f80e00 x1876699494081408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762788.908871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494081408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762788.908876:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762788.908878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762788.908880:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081408:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762788.908883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494081536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762788.908893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762788.908899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762788/real 1789762788] req@ffff8eb650f80700 x1876699494081536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762788.908906:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494081536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762788.908911:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762788.908913:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762788.908915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494081536:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762789.612535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762790.636534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762791.660551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762792.684550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762793.708579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762794.028464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762794.028473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762794.028477:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762794.028487:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2798 00000100:00080000:2.0:1789762794.028496:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762794.028502:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762794.028539:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494081664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762794.028572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762794.028580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762794.028582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762794.028588:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2798 00000100:00080000:2.0:1789762794.028593:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762794.028598:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762794.028607:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762794.028613:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494081664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762794.028620:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494081792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762794.028629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762794.028632:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762794.028634:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762794.028636:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2798 00000100:00080000:2.0:1789762794.028640:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762794.028642:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762794.028646:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762794.028653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494081920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762794.028661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762794.028663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762794.028665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762794.028669:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762794.028671:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2798 00000100:00080000:2.0:1789762794.028675:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762794.028678:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762794.028683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762794.028690:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494082048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762794.028717:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff8eb650f82680 x1876699494081664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762794.028733:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494081664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762794.028741:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762794.028744:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762794.028747:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494081664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762794.028765:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762794.028781:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762794.028789:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff8eb650f82d80 x1876699494081792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762794.028798:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494081792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762794.028803:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762794.028806:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762794.028808:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081792:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762794.028812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081920:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762794.028824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762794.028830:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff8eb650f81880 x1876699494081920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762794.028838:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494081920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762794.028843:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762794.028845:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762794.028847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494081920:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762794.028850:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494082048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762794.028860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762794.028877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762794/real 1789762794] req@ffff8eb650f81c00 x1876699494082048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762794.028885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494082048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762794.028890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762794.028891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762794.028893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494082048:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762794.732558:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762795.756561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762796.780521:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762797.804523:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762798.828525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762799.148494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762799.148502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762799.148506:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762799.148514:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2803 00000100:00080000:3.0:1789762799.148523:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762799.148528:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762799.148559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494082176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762799.148585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762799.148588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762799.148589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762799.148592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2803 00000100:00080000:3.0:1789762799.148595:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762799.148598:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762799.148606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762799.148615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494082304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762799.148623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762799.148625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762799.148627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762799.148632:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2803 00000100:00100000:1.0:1789762799.148633:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494082176:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762799.148634:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762799.148641:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762799.148645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762799.148658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494082432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762799.148669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762799.148673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762799.148675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762799.148680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2803 00000100:00080000:3.0:1789762799.148686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762799.148690:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762799.148696:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789762799.148700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762799.148716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494082560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762799.148733:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff8eb645149500 x1876699494082176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762799.148756:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494082176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762799.148764:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762799.148767:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762799.148770:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494082176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762799.148788:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762799.148803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762799.148811:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff8eb645148a80 x1876699494082304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762799.148819:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494082304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762799.148825:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762799.148827:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762799.148829:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082304:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762799.148833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082432:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762799.148856:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762799.148863:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff8eb64514a680 x1876699494082432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762799.148871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494082432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762799.148880:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762799.148882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762799.148884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082432:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762799.148888:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494082560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762799.148899:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762799.148904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762799/real 1789762799] req@ffff8eb645149c00 x1876699494082560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762799.148911:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494082560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762799.148916:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762799.148918:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762799.148920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494082560:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762799.852574:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762800.876540:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762801.900547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762802.924534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762803.948542:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762804.268504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.268513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762804.268517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.268526:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2808 00000100:00080000:1.0:1789762804.268539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.268547:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762804.268586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494082688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.268609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.268612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.268614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.268617:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2808 00000100:00080000:1.0:1789762804.268620:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.268624:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762804.268631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.268641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494082816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.268650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762804.268653:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494082688:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762804.268653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.268657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.268664:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2808 00000100:00080000:1.0:1789762804.268666:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.268668:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762804.268672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762804.268681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494082944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762804.268688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762804.268691:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762804.268692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762804.268695:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2808 00000100:00080000:1.0:1789762804.268707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762804.268710:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762804.268714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762804.268721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494083072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762804.268761:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082816:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762804.268793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082944:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762804.268814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494083072:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762804.972543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762805.996516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762807.020575:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762807.340613:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789762807.340649:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762807.340662:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789762807.340689:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789762807.340695:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789762807.340701:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762807.340737:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762807] req@ffff8eb65016d500 x1876699494082816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762807.340763:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494082816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762807.340774:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762807.340779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762807.340783:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082816:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762807.340791:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762807] req@ffff8eb65016fb80 x1876699494082944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762807.340803:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494082944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762807.340811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762807.340814:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762807.340817:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494082944:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762807.340824:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762807] req@ffff8eb65016c380 x1876699494083072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762807.340834:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494083072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762807.340841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762807.340844:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762807.340847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494083072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762807.340853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762804/real 1789762807] req@ffff8eb65016e300 x1876699494082688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762807.340863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494082688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762807.340870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762807.340873:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762807.340891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494082688:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762808.044527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762809.068582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762809.388541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762809.388555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762809.388560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762809.388571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2813 00000100:00080000:3.0:1789762809.388584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762809.388593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762809.388641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494083200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762809.388672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762809.388677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762809.388679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762809.388686:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2813 00000100:00080000:3.0:1789762809.388691:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762809.388698:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762809.388714:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762809.388727:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494083200:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762809.388737:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494083328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762809.388760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762809.388764:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762809.388766:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762809.388771:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2813 00000100:00080000:3.0:1789762809.388776:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762809.388782:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762809.388789:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762809.388802:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762809.388808:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494083456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762809.388823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762809.388829:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762809.388831:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762809.388838:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2813 00000100:00080000:3.0:1789762809.388843:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762809.388851:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff8eb645149c00 x1876699494083200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762809.388852:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762809.388882:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762809.388900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494083200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762809.388904:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494083584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762809.388969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762809.388978:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762809.388984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494083200:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762809.389021:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762809.389050:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762809.389064:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff8eb64514a680 x1876699494083328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762809.389078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494083328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762809.389087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762809.389091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762809.389095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083328:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762809.389102:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083456:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762809.389124:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762809.389137:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff8eb645148a80 x1876699494083456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762809.389148:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494083456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762809.389157:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762809.389161:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762809.389165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083456:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762809.389172:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494083584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762809.389193:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762809.389204:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762809/real 1789762809] req@ffff8eb645149500 x1876699494083584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762809.389215:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494083584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762809.389223:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762809.389226:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762809.389228:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494083584:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762810.092519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762811.116587:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762812.140535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762813.164582:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762814.188539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762814.508461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762814.508469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762814.508473:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762814.508482:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2819 00000100:00080000:3.0:1789762814.508492:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762814.508500:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762814.508537:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494083712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762814.508563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762814.508566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762814.508568:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762814.508573:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2819 00000100:00080000:3.0:1789762814.508577:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762814.508582:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762814.508592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762814.508606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494083840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762814.508611:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494083712:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762814.508617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762814.508621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762814.508622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762814.508626:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2819 00000100:00080000:3.0:1789762814.508629:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762814.508633:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762814.508637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762814.508646:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494083968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762814.508655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762814.508658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762814.508660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762814.508663:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2819 00000100:00080000:3.0:1789762814.508665:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762814.508668:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762814.508669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762814.508673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762814.508681:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494084096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762814.508705:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff8eb64514a300 x1876699494083712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762814.508723:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494083712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762814.508734:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762814.508738:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762814.508740:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494083712:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762814.508759:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762814.508777:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762814.508786:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff8eb645148380 x1876699494083840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762814.508794:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494083840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762814.508799:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762814.508801:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762814.508803:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083840:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762814.508807:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083968:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762814.508835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762814.508848:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff8eb645148e00 x1876699494083968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762814.508862:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494083968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762814.508871:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762814.508874:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762814.508878:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494083968:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762814.508884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494084096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762814.508900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762814.508907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762814/real 1789762814] req@ffff8eb64514b800 x1876699494084096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762814.508915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494084096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762814.508920:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762814.508922:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762814.508924:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494084096:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762815.212517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762816.236545:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762817.260569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762818.284513:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762819.308524:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762819.628478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.628486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762819.628489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.628497:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2824 00000100:00080000:1.0:1789762819.628506:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.628513:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762819.628549:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494084224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.628570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.628573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.628574:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.628577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2824 00000100:00080000:1.0:1789762819.628579:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.628582:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762819.628589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762819.628598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494084352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.628606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.628609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.628610:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.628613:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2824 00000100:00080000:1.0:1789762819.628614:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762819.628618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494084224:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762819.628619:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762819.628626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762819.628633:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494084480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762819.628642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762819.628645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762819.628646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762819.628650:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2824 00000100:00080000:1.0:1789762819.628652:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762819.628655:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762819.628658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762819.628666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494084608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762819.628679:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762819.628712:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff8eb65016ce00 x1876699494084224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762819.628725:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494084224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762819.628733:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762819.628736:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762819.628739:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494084224:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762819.628744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762819.628757:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762819.628764:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff8eb65016c700 x1876699494084352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762819.628772:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494084352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762819.628777:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762819.628779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762819.628796:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762819.628820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084480:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762819.628836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762819.628843:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff8eb65016df80 x1876699494084480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762819.628856:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494084480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762819.628862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762819.628864:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762819.628866:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084480:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762819.628869:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494084608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762819.628878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762819.628884:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762819/real 1789762819] req@ffff8eb65016dc00 x1876699494084608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762819.628891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494084608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762819.628896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762819.628898:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762819.628900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494084608:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762820.332508:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762821.356507:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762822.380514:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762823.404550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762824.428540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762824.748488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.748495:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762824.748499:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.748508:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2829 00000100:00080000:3.0:1789762824.748518:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.748524:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762824.748555:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494084736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.748578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.748582:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.748585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.748590:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2829 00000100:00080000:3.0:1789762824.748594:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.748598:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762824.748608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762824.748612:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494084736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762824.748620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494084864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.748630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762824.748633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.748635:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.748638:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2829 00000100:00080000:3.0:1789762824.748641:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.748644:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762824.748649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762824.748659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494084992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762824.748669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762824.748672:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762824.748672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762824.748674:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762824.748678:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2829 00000100:00080000:3.0:1789762824.748680:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762824.748683:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762824.748687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762824.748697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494085120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762824.748707:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762824] req@ffff8eb64514bb80 x1876699494084736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762824.748724:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494084736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762824.748735:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762824.748739:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762824.748755:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494084736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762824.748775:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762824.748795:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762824.748808:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762824] req@ffff8eb64514b100 x1876699494084864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762824.748821:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494084864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762824.748837:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762824.748841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762824.748845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084864:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762824.748851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084992:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762824.748868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762824.748876:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762824] req@ffff8eb645149880 x1876699494084992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762824.748884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494084992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762824.748890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762824.748892:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762824.748894:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494084992:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762824.748898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494085120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762824.748908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762824.748914:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762824/real 1789762824] req@ffff8eb645148000 x1876699494085120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762824.748921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494085120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762824.748926:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762824.748928:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762824.748930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494085120:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762825.452547:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762826.476514:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762827.500560:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762828.524591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762829.548531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762829.868470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762829.868480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762829.868484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762829.868494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2834 00000100:00080000:3.0:1789762829.868504:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762829.868511:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762829.868547:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494085248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762829.868573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762829.868577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762829.868579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762829.868584:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2834 00000100:00080000:3.0:1789762829.868588:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762829.868593:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762829.868602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762829.868611:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494085248:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789762829.868618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494085376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762829.868630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762829.868634:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762829.868636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762829.868640:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2834 00000100:00080000:3.0:1789762829.868645:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762829.868649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762829.868655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762829.868667:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494085504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762829.868677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762829.868682:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789762829.868684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762829.868688:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762829.868693:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2834 00000100:00080000:3.0:1789762829.868699:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762829.868703:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762829.868721:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762829.868737:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762829/real 1789762829] req@ffff8eb64514b480 x1876699494085248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762829.868739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494085632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762829.868775:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494085248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762829.868789:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762829.868794:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762829.868799:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494085248:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762829.868822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762829.868841:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762829.868850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762829/real 1789762829] req@ffff8eb645149180 x1876699494085376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762829.868859:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494085376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762829.868865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762829.868867:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762829.868869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085376:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762829.868873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085504:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762829.868885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762829.868891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762829/real 1789762829] req@ffff8eb64514ad80 x1876699494085504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762829.868898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494085504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762829.868903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762829.868905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762829.868907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085504:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762829.868910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494085632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762829.868920:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762829.868926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762829/real 1789762829] req@ffff8eb645148700 x1876699494085632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762829.868933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494085632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762829.868938:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762829.868940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762829.868942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494085632:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762830.572530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762831.596528:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762832.620547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762833.644520:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762834.668533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762834.988493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762834.988511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762834.988514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762834.988523:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2839 00000100:00080000:2.0:1789762834.988532:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762834.988538:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762834.988571:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494085760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762834.988593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762834.988596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762834.988597:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762834.988600:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2839 00000100:00080000:2.0:1789762834.988602:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762834.988605:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762834.988612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762834.988627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494085888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762834.988633:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494085760:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762834.988638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762834.988642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762834.988643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762834.988646:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2839 00000100:00080000:2.0:1789762834.988650:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762834.988652:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762834.988659:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762834.988681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494086016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762834.988690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762834.988693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762834.988694:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762834.988697:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2839 00000100:00080000:2.0:1789762834.988700:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762834.988703:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762834.988708:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789762834.988710:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762834.988722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494086144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762834.988752:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762834/real 1789762834] req@ffff8eb650f81c00 x1876699494085760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762834.988769:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494085760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762834.988778:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762834.988781:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762834.988784:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494085760:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762834.988802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762834.988819:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762834.988827:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762834/real 1789762834] req@ffff8eb650f81880 x1876699494085888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762834.988835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494085888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762834.988841:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762834.988843:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762834.988845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494085888:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762834.988849:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086016:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762834.988860:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762834.988866:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762834/real 1789762834] req@ffff8eb650f82d80 x1876699494086016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762834.988874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494086016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762834.988879:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762834.988881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762834.988883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086016:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762834.988887:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494086144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762834.988898:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762834.988904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762834/real 1789762834] req@ffff8eb650f82680 x1876699494086144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762834.988912:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494086144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762834.988917:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762834.988918:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762834.988920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494086144:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762835.692522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762836.716529:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762837.740525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762838.764519:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762839.788544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762840.108471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762840.108479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762840.108483:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762840.108491:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2844 00000100:00080000:2.0:1789762840.108500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762840.108505:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762840.108537:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494086272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762840.108561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762840.108566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762840.108567:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762840.108570:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2844 00000100:00080000:2.0:1789762840.108572:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762840.108575:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762840.108582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762840.108590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494086400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762840.108598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762840.108600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762840.108601:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762840.108604:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2844 00000100:00080000:2.0:1789762840.108606:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762840.108608:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762840.108612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762840.108617:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494086272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762840.108622:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494086528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762840.108646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762840.108649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762840.108650:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762840.108653:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2844 00000100:00080000:2.0:1789762840.108656:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762840.108659:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762840.108664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762840.108675:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494086656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762840.108697:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762840.108737:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff8eb650f80700 x1876699494086272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762840.108754:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494086272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762840.108764:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762840.108768:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762840.108772:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494086272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762840.108780:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762840.108803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762840.108814:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff8eb650f80e00 x1876699494086400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762840.108827:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494086400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762840.108836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762840.108839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762840.108842:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086400:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762840.108866:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086528:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762840.108885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762840.108895:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff8eb650f83480 x1876699494086528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762840.108907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494086528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762840.108915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762840.108918:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762840.108930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086528:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762840.108937:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494086656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762840.108955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762840.108966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762840/real 1789762840] req@ffff8eb650f82a00 x1876699494086656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762840.108978:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494086656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762840.108991:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762840.108995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762840.108998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494086656:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762840.812562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762841.836599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762842.860592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762843.884499:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762844.908539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762845.228483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.228492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762845.228495:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.228505:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2849 00000100:00080000:1.0:1789762845.228514:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.228520:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762845.228553:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494086784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.228577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.228580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.228581:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.228584:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2849 00000100:00080000:1.0:1789762845.228587:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.228603:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762845.228612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762845.228619:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494086784:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762845.228630:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494086912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.228644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.228648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.228651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.228654:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2849 00000100:00080000:1.0:1789762845.228666:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762845.228669:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762845.228675:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762845.228675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762845.228688:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494087040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762845.228696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762845.228698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762845.228699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762845.228703:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2849 00000100:00080000:1.0:1789762845.228706:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762845.228711:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff8eb65016f480 x1876699494086784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762845.228711:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762845.228718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762845.228730:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494087168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762845.228735:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494086784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762845.228748:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762845.228752:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762845.228755:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494086784:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762845.228775:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762845.228789:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762845.228797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff8eb65016ed80 x1876699494086912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762845.228805:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494086912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762845.228811:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762845.228813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762845.228815:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494086912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762845.228819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087040:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762845.228830:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762845.228835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff8eb65016ca80 x1876699494087040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762845.228842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494087040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762845.228847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762845.228849:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762845.228851:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087040:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762845.228854:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494087168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762845.228864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762845.228870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762845/real 1789762845] req@ffff8eb65016d180 x1876699494087168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762845.228877:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494087168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762845.228882:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762845.228883:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762845.228885:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494087168:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762845.932515:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762846.956521:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762847.980546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762849.004535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762850.028525:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762850.348446:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762850.348454:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762850.348457:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762850.348465:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2854 00000100:00080000:1.0:1789762850.348474:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762850.348479:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762850.348510:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494087296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762850.348535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762850.348538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762850.348539:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762850.348542:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2854 00000100:00080000:1.0:1789762850.348544:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762850.348547:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762850.348566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762850.348578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494087424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762850.348584:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494087296:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762850.348590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762850.348594:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762850.348596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762850.348599:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2854 00000100:00080000:1.0:1789762850.348601:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762850.348604:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762850.348608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762850.348617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494087552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762850.348624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762850.348627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762850.348628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762850.348630:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2854 00000100:00080000:1.0:1789762850.348635:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762850.348638:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762850.348641:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762850.348651:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762850.348651:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494087680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762850.348689:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff8eb65016c000 x1876699494087296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762850.348703:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494087296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762850.348711:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762850.348714:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762850.348716:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494087296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762850.348736:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762850.348751:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762850.348759:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff8eb65016f100 x1876699494087424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762850.348768:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494087424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762850.348773:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762850.348775:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762850.348777:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087424:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762850.348781:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087552:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762850.348791:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762850.348797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff8eb65016d180 x1876699494087552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762850.348804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494087552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762850.348809:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762850.348811:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762850.348813:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087552:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762850.348816:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494087680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762850.348826:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762850.348832:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762850/real 1789762850] req@ffff8eb65016ca80 x1876699494087680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762850.348839:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494087680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762850.348844:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762850.348846:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762850.348847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494087680:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762851.052517:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762852.076576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762853.100591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762854.124584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762855.148558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762855.468488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762855.468498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762855.468503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762855.468513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2859 00000100:00080000:1.0:1789762855.468537:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762855.468545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762855.468584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494087808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762855.468607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762855.468612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762855.468614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762855.468618:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2859 00000100:00080000:1.0:1789762855.468622:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762855.468627:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762855.468637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762855.468648:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494087808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762855.468650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494087936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762855.468661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762855.468664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762855.468666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762855.468669:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2859 00000100:00080000:1.0:1789762855.468672:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762855.468675:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762855.468680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762855.468690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494088064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762855.468698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762855.468702:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762855.468703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762855.468706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2859 00000800:00000100:0.0:1789762855.468709:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762855.468709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762855.468713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762855.468717:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762855.468726:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494088192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762855.468758:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff8eb65016ed80 x1876699494087808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.468783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494087808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.468791:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.468794:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762855.468797:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494087808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762855.468818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762855.468837:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762855.468846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff8eb65016f480 x1876699494087936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.468854:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494087936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.468859:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.468861:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762855.468863:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494087936:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762855.468867:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088064:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762855.468878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762855.468883:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff8eb65016dc00 x1876699494088064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.468891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494088064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.468900:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.468901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762855.468903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088064:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762855.468906:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494088192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762855.468916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762855.468922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762855/real 1789762855] req@ffff8eb65016df80 x1876699494088192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762855.468928:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494088192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762855.468933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762855.468935:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762855.468937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494088192:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762856.172532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762857.196533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762858.220523:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762859.244599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762860.268554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762860.588554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.588568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762860.588573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.588585:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2865 00000100:00080000:3.0:1789762860.588596:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.588603:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789762860.588641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494088320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762860.588669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.588672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762860.588674:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.588678:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2865 00000100:00080000:3.0:1789762860.588681:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.588684:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762860.588692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762860.588705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494088448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762860.588715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.588720:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762860.588723:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494088320:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789762860.588723:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.588734:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2865 00000100:00080000:3.0:1789762860.588737:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.588740:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762860.588744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762860.588757:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494088576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762860.588781:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762860.588785:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762860.588786:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762860.588790:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2865 00000100:00080000:3.0:1789762860.588795:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762860.588798:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789762860.588802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762860.588808:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789762860.588818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494088704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762860.588854:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff8eb645148700 x1876699494088320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.588872:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494088320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.588888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.588891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762860.588895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494088320:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762860.588917:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762860.588947:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762860.588960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff8eb64514ad80 x1876699494088448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.588971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494088448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.588978:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.588981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762860.588984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088448:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762860.588989:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088576:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762860.589005:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762860.589013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff8eb645149180 x1876699494088576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.589022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494088576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.589029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.589031:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762860.589033:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088576:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762860.589039:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494088704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762860.589061:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762860.589069:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762860/real 1789762860] req@ffff8eb64514b480 x1876699494088704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762860.589079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494088704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762860.589086:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762860.589088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762860.589091:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494088704:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762861.292540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762862.316569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762863.340525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762864.364558:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762865.388563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762865.708510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762865.708517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762865.708521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762865.708530:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2870 00000100:00080000:1.0:1789762865.708539:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762865.708545:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762865.708581:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494088832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762865.708601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762865.708604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762865.708605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762865.708609:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2870 00000100:00080000:1.0:1789762865.708611:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762865.708615:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762865.708623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762865.708632:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494088832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762865.708635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494088960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762865.708646:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762865.708648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762865.708649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762865.708652:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2870 00000100:00080000:1.0:1789762865.708654:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762865.708657:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762865.708660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762865.708668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494089088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762865.708675:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762865.708677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762865.708679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762865.708682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2870 00000100:00080000:1.0:1789762865.708685:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762865.708689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789762865.708690:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789762865.708695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762865.708704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494089216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762865.708732:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff8eb65016c700 x1876699494088832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762865.708749:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494088832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762865.708762:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762865.708766:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762865.708768:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494088832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762865.708786:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762865.708803:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762865.708811:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff8eb65016ce00 x1876699494088960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762865.708819:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494088960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762865.708825:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762865.708827:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762865.708829:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494088960:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762865.708833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089088:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762865.708844:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762865.708850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff8eb65016e300 x1876699494089088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762865.708857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494089088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762865.708863:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762865.708864:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762865.708866:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089088:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762865.708870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494089216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762865.708880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762865.708886:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762865/real 1789762865] req@ffff8eb65016c380 x1876699494089216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762865.708898:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494089216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762865.708903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762865.708905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762865.708907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494089216:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762866.412556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762867.436568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762868.460690:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762869.484748:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762870.508554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762870.828542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762870.828552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762870.828557:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762870.828568:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2875 00000100:00080000:2.0:1789762870.828580:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762870.828587:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762870.828632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494089344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762870.828659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762870.828663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762870.828665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762870.828670:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2875 00000100:00080000:2.0:1789762870.828675:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762870.828681:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762870.828693:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762870.828710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494089344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762870.828721:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494089472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762870.828732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762870.828735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762870.828737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762870.828741:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2875 00000100:00080000:2.0:1789762870.828745:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762870.828749:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762870.828755:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762870.828767:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494089600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762870.828776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762870.828779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762870.828781:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762870.828783:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2875 00000100:00080000:2.0:1789762870.828786:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762870.828789:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762870.828792:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762870.828802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494089728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762870.828843:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089472:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762870.828874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089600:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762870.828895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494089728:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762871.532604:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762872.556557:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762873.580586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762873.900676:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762873.900723:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762873.900740:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762873.900785:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762873.900802:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762873.900812:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762873.900858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762873] req@ffff8eb650f81180 x1876699494089472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762873.900890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494089472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762873.900906:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762873.900912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762873.900917:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089472:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762873.900927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762873] req@ffff8eb650f83b80 x1876699494089600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762873.900941:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494089600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762873.900949:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762873.900952:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762873.900956:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089600:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762873.900963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762873] req@ffff8eb650f81f80 x1876699494089728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762873.900975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494089728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762873.900983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762873.900987:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762873.900990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494089728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762873.900997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762870/real 1789762873] req@ffff8eb650f83800 x1876699494089344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762873.901017:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494089344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762873.901025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762873.901029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762873.901033:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494089344:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789762874.604586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762875.628591:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762875.948533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762875.948541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762875.948545:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762875.948554:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2880 00000100:00080000:1.0:1789762875.948563:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762875.948569:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762875.948597:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494089856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762875.948619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762875.948622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762875.948624:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762875.948626:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2880 00000100:00080000:1.0:1789762875.948628:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762875.948632:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762875.948640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762875.948657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494089984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762875.948667:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494089856:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762875.948673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762875.948681:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762875.948685:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762875.948690:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2880 00000100:00080000:1.0:1789762875.948694:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762875.948699:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762875.948705:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762875.948720:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494090112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762875.948732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762875.948735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762875.948737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762875.948743:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762875.948746:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2880 00000100:00080000:1.0:1789762875.948752:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762875.948757:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762875.948763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762875.948776:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494090240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762875.948788:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762875/real 1789762875] req@ffff8eb65016fb80 x1876699494089856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762875.948809:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494089856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762875.948821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762875.948825:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762875.948830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494089856:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762875.948853:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762875.948878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762875.948891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762875/real 1789762875] req@ffff8eb65016d500 x1876699494089984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762875.948904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494089984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762875.948913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762875.948916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762875.948920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494089984:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762875.948926:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090112:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762875.948949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762875.948961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762875/real 1789762875] req@ffff8eb65016d880 x1876699494090112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762875.948973:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494090112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762875.948981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762875.948985:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762875.948988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090112:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762875.948994:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494090240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762875.949011:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762875.949022:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762875/real 1789762875] req@ffff8eb65016ea00 x1876699494090240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762875.949034:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494090240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762875.949042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762875.949046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762875.949050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494090240:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762876.652649:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762877.676586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762878.700574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762879.724589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762880.748512:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762881.068478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762881.068486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762881.068490:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762881.068499:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2885 00000100:00080000:2.0:1789762881.068508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762881.068514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762881.068550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494090368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762881.068578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762881.068585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762881.068586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762881.068590:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2885 00000100:00080000:2.0:1789762881.068593:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762881.068596:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762881.068604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762881.068614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494090496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762881.068622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762881.068625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762881.068626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762881.068630:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2885 00000100:00080000:2.0:1789762881.068632:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762881.068635:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762881.068638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762881.068646:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494090624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762881.068654:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494090368:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762881.068662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762881.068668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762881.068670:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762881.068675:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2885 00000100:00080000:2.0:1789762881.068680:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762881.068685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762881.068692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762881.068704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494090752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762881.068714:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762881.068746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff8eb650f80a80 x1876699494090368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762881.068759:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494090368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762881.068767:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762881.068770:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762881.068773:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494090368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762881.068778:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762881.068790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762881.068797:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff8eb650f80000 x1876699494090496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762881.068805:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494090496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762881.068810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762881.068813:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762881.068815:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090496:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762881.068831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090624:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762881.068841:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762881.068847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff8eb650f80380 x1876699494090624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762881.068854:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494090624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762881.068864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762881.068866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762881.068869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494090624:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762881.068872:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494090752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762881.068881:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762881.068886:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762881/real 1789762881] req@ffff8eb650f81500 x1876699494090752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762881.068893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494090752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762881.068898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762881.068900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762881.068902:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494090752:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762881.772516:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762882.796525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762883.820569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762884.844521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762885.868495:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762886.188497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762886.188508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762886.188511:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762886.188521:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2890 00000100:00080000:2.0:1789762886.188532:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762886.188539:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762886.188579:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494090880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762886.188607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762886.188610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762886.188612:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762886.188616:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2890 00000100:00080000:2.0:1789762886.188619:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762886.188623:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762886.188632:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762886.188648:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494091008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762886.188650:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494090880:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762886.188669:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762886.188672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762886.188673:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762886.188677:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2890 00000100:00080000:2.0:1789762886.188681:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762886.188685:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762886.188690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762886.188703:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494091136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762886.188713:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762886.188728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762886.188729:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789762886.188731:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762886.188739:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2890 00000100:00080000:2.0:1789762886.188744:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762886.188749:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762886.188754:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762886.188767:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494091264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762886.188778:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff8eb650f81500 x1876699494090880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762886.188811:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494090880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762886.188829:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762886.188833:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762886.188836:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494090880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762886.188858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762886.188879:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762886.188890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff8eb650f80380 x1876699494091008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762886.188901:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494091008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762886.188909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762886.188912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762886.188914:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762886.188919:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091136:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762886.188934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762886.188942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff8eb650f80000 x1876699494091136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762886.188952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494091136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762886.188959:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762886.188961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762886.188964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091136:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762886.188969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494091264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762886.188982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762886.188990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762886/real 1789762886] req@ffff8eb650f80a80 x1876699494091264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762886.189000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494091264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762886.189006:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762886.189009:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762886.189011:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494091264:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762886.892573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762887.916550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762888.940603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762889.964516:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762890.988526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762891.308484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762891.308493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762891.308497:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762891.308505:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2895 00000100:00080000:1.0:1789762891.308515:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762891.308521:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762891.308556:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494091392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762891.308578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762891.308581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762891.308583:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762891.308586:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2895 00000100:00080000:1.0:1789762891.308588:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762891.308591:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762891.308599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762891.308609:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494091520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762891.308618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494091392:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762891.308620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762891.308627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762891.308629:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762891.308632:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2895 00000100:00080000:1.0:1789762891.308634:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762891.308637:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762891.308641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762891.308650:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494091648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762891.308658:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762891.308661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762891.308662:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762891.308665:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2895 00000100:00080000:1.0:1789762891.308669:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762891.308672:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762891.308675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762891.308683:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762891.308684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494091776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762891.308723:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762891] req@ffff8eb65016e680 x1876699494091392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762891.308738:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494091392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762891.308747:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762891.308750:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762891.308753:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494091392:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762891.308774:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762891.308790:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762891.308799:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762891] req@ffff8eb65016f800 x1876699494091520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762891.308807:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494091520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762891.308813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762891.308816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762891.308818:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091520:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762891.308822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091648:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762891.308832:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762891.308839:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762891] req@ffff8eb65016ea00 x1876699494091648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762891.308846:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494091648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762891.308852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762891.308854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762891.308856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494091648:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762891.308860:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494091776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762891.308870:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762891.308877:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762891/real 1789762891] req@ffff8eb65016d880 x1876699494091776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762891.308885:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494091776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762891.308903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762891.308906:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762891.308910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494091776:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762892.012614:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762893.036594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762894.060524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762895.084623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762896.108617:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762896.428537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762896.428549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762896.428555:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762896.428580:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2900 00000100:00080000:0.0:1789762896.428594:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762896.428603:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762896.428651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494091904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762896.428686:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762896.428691:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762896.428694:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762896.428699:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2900 00000100:00080000:0.0:1789762896.428703:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762896.428707:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762896.428719:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762896.428735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494092032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762896.428751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762896.428756:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762896.428759:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762896.428768:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494091904:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762896.428770:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2900 00000100:00080000:0.0:1789762896.428780:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762896.428788:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762896.428795:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762896.428813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494092160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762896.428827:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762896.428831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762896.428834:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762896.428839:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2900 00000100:00080000:0.0:1789762896.428845:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762896.428850:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762896.428856:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762896.428865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762896.428874:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494092288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762896.428930:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff8eb64304d500 x1876699494091904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.428955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494091904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.428968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.428973:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762896.428978:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494091904:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762896.429006:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762896.429033:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762896.429063:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff8eb64304d880 x1876699494092032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.429079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494092032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.429091:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.429095:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762896.429099:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092032:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762896.429106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092160:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762896.429129:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762896.429141:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff8eb64304f480 x1876699494092160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.429155:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494092160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.429165:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.429169:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762896.429173:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092160:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762896.429180:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494092288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762896.429198:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762896.429209:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762896/real 1789762896] req@ffff8eb64304e680 x1876699494092288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762896.429252:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494092288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762896.429271:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762896.429275:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762896.429280:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494092288:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762897.132599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762898.156508:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762899.180506:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762900.204530:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762901.228593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762901.548508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762901.548521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762901.548526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762901.548538:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2906 00000100:00080000:1.0:1789762901.548550:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762901.548559:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762901.548598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494092416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762901.548633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762901.548637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762901.548640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762901.548647:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2906 00000100:00080000:1.0:1789762901.548653:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762901.548659:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762901.548669:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494092416:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762901.548671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762901.548708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494092544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762901.548726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762901.548731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1789762901.548735:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762901.548737:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762901.548746:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2906 00000100:00080000:1.0:1789762901.548752:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762901.548757:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762901.548765:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762901.548781:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494092672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762901.548785:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff8eb65016d500 x1876699494092416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762901.548813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494092416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762901.548828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762901.548830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762901.548838:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762901.548839:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:1.0:1789762901.548846:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762901.548847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494092416:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762901.548858:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2906 00000100:00080000:1.0:1789762901.548865:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762901.548870:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762901.548876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092544:192.168.203.160@tcp:38:kworker.0 00000008:00100000:1.0:1789762901.548878:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762901.548902:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494092800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762901.548907:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762901.548922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff8eb65016fb80 x1876699494092544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762901.548935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494092544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762901.548943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762901.548947:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762901.548950:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092544:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762901.548955:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092672:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762901.548968:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762901.548975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff8eb65016c380 x1876699494092672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762901.548989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494092672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762901.548995:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762901.548997:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762901.548999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494092672:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762901.549009:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494092800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762901.549021:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762901.549028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762901/real 1789762901] req@ffff8eb65016e300 x1876699494092800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762901.549036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494092800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762901.549042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762901.549044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762901.549046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494092800:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762902.252566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762903.276510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762904.300590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762905.324546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762906.348523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762906.668465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.668473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762906.668476:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.668484:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2911 00000100:00080000:1.0:1789762906.668493:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.668499:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762906.668532:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494092928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.668558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.668561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.668562:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.668565:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2911 00000100:00080000:1.0:1789762906.668567:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.668569:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762906.668577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.668585:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494093056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.668593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.668596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.668597:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.668599:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2911 00000100:00080000:1.0:1789762906.668601:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.668603:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762906.668607:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762906.668617:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494092928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762906.668617:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494093184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762906.668631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762906.668633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762906.668634:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762906.668637:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2911 00000100:00080000:1.0:1789762906.668639:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762906.668642:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762906.668645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762906.668654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494093312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762906.668683:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762906.668725:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff8eb65016ce00 x1876699494092928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762906.668745:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494092928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762906.668752:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762906.668755:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762906.668759:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494092928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762906.668764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762906.668780:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762906.668788:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff8eb65016c700 x1876699494093056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762906.668796:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494093056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762906.668802:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762906.668803:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762906.668806:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093056:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762906.668825:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093184:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762906.668836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762906.668841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff8eb65016df80 x1876699494093184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762906.668849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494093184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762906.668858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762906.668860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762906.668862:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093184:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762906.668865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494093312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762906.668874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762906.668880:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762906/real 1789762906] req@ffff8eb65016dc00 x1876699494093312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762906.668887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494093312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762906.668892:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762906.668893:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762906.668895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494093312:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762907.372530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762908.396519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762909.420554:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762910.444566:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762911.468621:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762911.788492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762911.788500:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762911.788503:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762911.788512:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2916 00000100:00080000:1.0:1789762911.788521:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762911.788527:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762911.788561:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494093440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762911.788590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762911.788594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762911.788597:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762911.788601:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2916 00000100:00080000:1.0:1789762911.788606:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762911.788610:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762911.788620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762911.788632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494093568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762911.788640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762911.788644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762911.788645:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762911.788648:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2916 00000100:00080000:1.0:1789762911.788653:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762911.788655:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762911.788659:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762911.788673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494093696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762911.788674:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494093440:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762911.788700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762911.788703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762911.788704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762911.788709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2916 00000100:00080000:1.0:1789762911.788711:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762911.788714:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762911.788717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762911.788725:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494093824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762911.788773:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762911.788819:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff8eb65016f480 x1876699494093440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762911.788835:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494093440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762911.788850:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762911.788853:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762911.788857:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494093440:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762911.788862:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762911.788878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762911.788886:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff8eb65016ed80 x1876699494093568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762911.788894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494093568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762911.788899:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762911.788901:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762911.788903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762911.788930:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093696:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762911.788942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762911.788949:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff8eb65016ca80 x1876699494093696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762911.788956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494093696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762911.788962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762911.788964:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762911.788965:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494093696:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762911.788970:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494093824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762911.788981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762911.788987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762911/real 1789762911] req@ffff8eb65016d180 x1876699494093824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762911.788995:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494093824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762911.788999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762911.789001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762911.789003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494093824:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762912.492625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762913.516590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762914.540634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762915.564617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762916.588621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762916.908598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762916.908609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762916.908614:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762916.908626:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2921 00000100:00080000:0.0:1789762916.908640:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762916.908648:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762916.908698:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494093952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762916.908730:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762916.908734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762916.908735:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762916.908739:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2921 00000100:00080000:0.0:1789762916.908742:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762916.908790:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762916.908803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762916.908831:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494094080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762916.908834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494093952:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762916.908844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762916.908848:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762916.908851:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762916.908856:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2921 00000100:00080000:0.0:1789762916.908860:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762916.908865:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762916.908870:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762916.908883:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494094208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762916.908891:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762916.908894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762916.908896:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762916.908899:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2921 00000100:00080000:0.0:1789762916.908916:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762916.908920:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762916.908924:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762916.908939:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494094336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762916.908940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762916.909008:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762916/real 1789762916] req@ffff8eb64304ea00 x1876699494093952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762916.909030:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494093952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762916.909051:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762916.909056:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762916.909087:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494093952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762916.909206:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762916.909246:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762916.909265:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762916/real 1789762916] req@ffff8eb64304c700 x1876699494094080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762916.909292:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494094080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762916.909308:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762916.909315:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762916.909322:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094080:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762916.909336:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094208:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762916.909379:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762916.909451:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762916/real 1789762916] req@ffff8eb64304f100 x1876699494094208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762916.909475:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494094208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762916.909507:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762916.909514:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762916.909521:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094208:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762916.909535:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494094336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762916.909577:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762916.909594:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762916/real 1789762916] req@ffff8eb64304ca80 x1876699494094336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762916.909610:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494094336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762916.909620:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762916.909624:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762916.909628:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494094336:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762917.612620:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762918.636594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762919.660642:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762920.684693:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762921.708558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762922.028522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762922.028533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762922.028538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762922.028549:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2926 00000100:00080000:2.0:1789762922.028563:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762922.028571:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762922.028615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494094464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762922.028641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762922.028647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762922.028648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762922.028651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2926 00000100:00080000:2.0:1789762922.028654:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762922.028656:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762922.028666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762922.028676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494094592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762922.028685:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762922.028689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762922.028690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762922.028693:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2926 00000100:00080000:2.0:1789762922.028695:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762922.028698:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762922.028701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762922.028708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494094720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762922.028717:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762922.028721:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762922.028724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762922.028733:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2926 00000100:00100000:1.0:1789762922.028735:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494094464:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762922.028737:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762922.028751:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762922.028759:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762922.028792:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494094848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762922.028873:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762922.028940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff8eb650f83800 x1876699494094464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762922.028967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494094464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762922.028982:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762922.028987:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762922.028992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494094464:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762922.029002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762922.029080:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762922.029095:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff8eb650f81f80 x1876699494094592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762922.029110:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494094592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762922.029119:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762922.029123:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762922.029126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094592:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762922.029154:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094720:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762922.029175:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762922.029185:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff8eb650f83b80 x1876699494094720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762922.029197:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494094720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762922.029205:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762922.029208:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762922.029210:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494094720:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762922.029216:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494094848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762922.029231:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762922.029239:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762922/real 1789762922] req@ffff8eb650f81180 x1876699494094848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762922.029250:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494094848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762922.029370:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762922.029378:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762922.029386:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494094848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762922.732576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762923.756574:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762924.780559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762925.804560:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762926.828550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762927.148524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762927.148534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762927.148539:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762927.148551:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2931 00000100:00080000:2.0:1789762927.148563:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762927.148572:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762927.148619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494094976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762927.148654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762927.148658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762927.148660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762927.148665:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2931 00000100:00080000:2.0:1789762927.148669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762927.148673:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762927.148683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762927.148694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494095104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762927.148702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762927.148705:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762927.148707:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762927.148713:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2931 00000100:00080000:2.0:1789762927.148723:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762927.148726:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762927.148731:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762927.148744:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494095232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762927.148745:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494094976:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762927.148770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762927.148773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762927.148776:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762927.148780:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2931 00000100:00080000:2.0:1789762927.148785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762927.148788:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762927.148793:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762927.148801:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494095360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762927.148829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762927.148880:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff8eb650f82a00 x1876699494094976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762927.148896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494094976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762927.148905:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762927.148908:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762927.148911:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494094976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762927.148916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762927.148932:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762927.148939:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff8eb650f83480 x1876699494095104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762927.148948:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494095104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762927.148953:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762927.148955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762927.148958:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762927.148979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095232:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762927.149005:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762927.149013:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff8eb650f80e00 x1876699494095232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762927.149020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494095232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762927.149026:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762927.149029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762927.149031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762927.149035:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494095360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762927.149046:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762927.149051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762927/real 1789762927] req@ffff8eb650f80700 x1876699494095360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762927.149059:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494095360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762927.149063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762927.149065:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762927.149067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494095360:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762927.852578:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762928.876589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762929.900628:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762930.924535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762931.948644:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762932.268578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762932.268595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762932.268602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762932.268621:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2936 00000100:00080000:2.0:1789762932.268642:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762932.268656:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762932.268723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494095488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762932.268779:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762932.268788:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762932.268792:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762932.268798:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2936 00000100:00080000:2.0:1789762932.268804:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762932.268813:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762932.268819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494095488:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789762932.268831:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762932.268853:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494095616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762932.268869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762932.268874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762932.268878:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762932.268883:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2936 00000100:00080000:2.0:1789762932.268892:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762932.268897:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789762932.268909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789762932.268914:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762932.268937:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494095744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762932.268968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff8eb650f82680 x1876699494095488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.268978:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762932.268985:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762932.268988:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762932.268996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494095488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762932.269024:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2936 00000100:00080000:1.0:1789762932.269025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762932.269030:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762932.269037:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:2.0:1789762932.269039:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762932.269053:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762932.269054:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494095488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789762932.269079:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494095872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762932.269106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762932.269144:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762932.269160:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff8eb650f82d80 x1876699494095616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762932.269174:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494095616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762932.269184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762932.269187:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762932.269191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095616:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762932.269198:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095744:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762932.269216:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762932.269224:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff8eb650f81880 x1876699494095744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762932.269232:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494095744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762932.269254:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762932.269258:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762932.269261:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494095744:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762932.269268:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494095872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762932.269291:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762932.269302:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762932/real 1789762932] req@ffff8eb650f81c00 x1876699494095872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762932.269316:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494095872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762932.269323:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762932.269325:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762932.269328:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494095872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762932.972644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762933.996674:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762935.020628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762936.044697:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762937.068597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762937.388530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.388542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762937.388546:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.388559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2941 00000100:00080000:1.0:1789762937.388572:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.388580:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762937.388629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494096000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.388659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.388663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762937.388664:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.388668:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2941 00000100:00080000:1.0:1789762937.388670:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.388673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762937.388683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762937.388692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494096128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.388700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.388702:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762937.388704:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.388706:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2941 00000100:00080000:1.0:1789762937.388707:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.388710:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762937.388713:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762937.388723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494096256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762937.388729:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762937.388732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762937.388734:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762937.388737:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2941 00000100:00080000:1.0:1789762937.388740:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762937.388745:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762937.388747:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494096000:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789762937.388750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762937.388761:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494096384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762937.388868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762937.388895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096256:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762937.388936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494096384:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762938.092633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762939.116559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762940.140539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762940.460690:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:1.0:1789762940.460743:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762940.460764:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762940.460816:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762940.460826:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:1.0:1789762940.460836:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789762940.460937:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762940] req@ffff8eb65016ca80 x1876699494096384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762940.460967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494096384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762940.460988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762940.460994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762940.460999:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494096384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762940.461008:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762940] req@ffff8eb65016f100 x1876699494096000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762940.461021:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494096000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762940.461029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762940.461032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762940.461036:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494096000:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762940.461044:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762940] req@ffff8eb65016c000 x1876699494096128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762940.461054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494096128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762940.461063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762940.461068:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762940.461072:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762940.461079:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762937/real 1789762940] req@ffff8eb65016d180 x1876699494096256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762940.461092:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494096256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762940.461100:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762940.461104:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762940.461107:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096256:192.168.203.160@tcp:8:kworker.0 00010000:00010000:3.0:1789762941.164545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762942.188580:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762942.508541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762942.508552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762942.508556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762942.508567:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2947 00000100:00080000:1.0:1789762942.508579:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762942.508586:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762942.508623:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494096512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762942.508655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762942.508662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762942.508664:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762942.508670:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2947 00000100:00080000:1.0:1789762942.508674:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762942.508683:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762942.508697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762942.508715:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494096640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762942.508727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762942.508732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762942.508734:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762942.508739:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2947 00000100:00080000:1.0:1789762942.508744:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762942.508750:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494096512:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762942.508754:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762942.508768:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762942.508792:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494096768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762942.508808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762942.508813:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762942.508815:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762942.508820:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2947 00000100:00080000:1.0:1789762942.508837:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762942.508843:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762942.508849:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762942.508855:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762942.508888:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494096896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762942.508943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff8eb65016ed80 x1876699494096512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762942.508984:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494096512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762942.508996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762942.509001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762942.509005:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494096512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762942.509014:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762942.509042:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762942.509056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff8eb65016f480 x1876699494096640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762942.509070:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494096640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762942.509079:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762942.509083:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762942.509088:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762942.509138:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096768:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762942.509164:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762942.509178:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff8eb65016dc00 x1876699494096768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762942.509191:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494096768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762942.509201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762942.509204:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762942.509207:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494096768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762942.509214:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494096896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762942.509241:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762942.509254:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762942/real 1789762942] req@ffff8eb65016df80 x1876699494096896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762942.509266:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494096896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762942.509274:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762942.509277:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762942.509280:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494096896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762943.212597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762944.236636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762945.260572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762946.284559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762947.308580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762947.628517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762947.628528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762947.628533:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762947.628543:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2952 00000100:00080000:2.0:1789762947.628554:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762947.628562:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762947.628595:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494097024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762947.628617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762947.628620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762947.628621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762947.628624:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2952 00000100:00080000:2.0:1789762947.628627:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762947.628631:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762947.628639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762947.628649:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494097152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762947.628654:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494097024:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762947.628660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762947.628664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762947.628665:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762947.628668:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2952 00000100:00080000:2.0:1789762947.628670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762947.628674:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762947.628678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762947.628687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494097280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762947.628694:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762947.628697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762947.628698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762947.628701:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2952 00000100:00080000:2.0:1789762947.628706:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762947.628708:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762947.628711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762947.628715:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762947.628720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494097408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762947.628753:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff8eb650f81c00 x1876699494097024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762947.628766:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494097024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762947.628774:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762947.628776:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762947.628779:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494097024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762947.628796:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762947.628810:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762947.628818:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff8eb650f81880 x1876699494097152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762947.628825:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494097152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762947.628830:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762947.628852:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762947.628855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762947.628859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097280:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762947.628874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762947.628882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff8eb650f82d80 x1876699494097280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762947.628890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494097280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762947.628895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762947.628897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762947.628900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762947.628903:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494097408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762947.628912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762947.628918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762947/real 1789762947] req@ffff8eb650f82680 x1876699494097408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762947.628925:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494097408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762947.628930:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762947.628931:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762947.628933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494097408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762948.332605:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762949.356590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762950.380595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762951.404570:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762952.428639:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762952.748484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.748492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762952.748495:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.748504:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2957 00000100:00080000:2.0:1789762952.748513:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.748519:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762952.748550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494097536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762952.748572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.748577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.748578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.748581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2957 00000100:00080000:2.0:1789762952.748583:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.748586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762952.748592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762952.748600:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494097664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762952.748607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.748609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.748611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.748612:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2957 00000100:00080000:2.0:1789762952.748614:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.748616:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762952.748619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762952.748624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494097792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762952.748630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762952.748632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762952.748634:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762952.748638:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2957 00000100:00080000:2.0:1789762952.748640:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762952.748642:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762952.748645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762952.748651:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494097920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762952.748658:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494097536:192.168.203.160@tcp:250:kworker.0 00000800:00000100:0.0:1789762952.748718:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762952.748759:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff8eb650f80700 x1876699494097536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762952.748786:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494097536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762952.748813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762952.748817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762952.748820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494097536:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762952.748826:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762952.748849:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762952.748862:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff8eb650f80e00 x1876699494097664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762952.748875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494097664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762952.748884:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762952.748888:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762952.748891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762952.748897:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097792:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762952.748914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762952.748922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff8eb650f83480 x1876699494097792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762952.748930:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494097792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762952.748939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762952.748942:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762952.748944:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494097792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762952.748964:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494097920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762952.748975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762952.748982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762952/real 1789762952] req@ffff8eb650f82a00 x1876699494097920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762952.748989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494097920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762952.748994:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762952.748996:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762952.748998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494097920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762953.452591:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762954.476519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762955.500631:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762956.524638:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762957.548501:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762957.868495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762957.868506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762957.868510:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762957.868521:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2962 00000100:00080000:2.0:1789762957.868549:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762957.868558:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762957.868616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494098048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762957.868647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762957.868651:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762957.868654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762957.868658:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2962 00000100:00080000:2.0:1789762957.868662:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762957.868666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762957.868676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762957.868690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494098176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762957.868702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762957.868707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762957.868709:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762957.868712:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2962 00000100:00080000:2.0:1789762957.868717:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762957.868725:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494098048:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762957.868726:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762957.868737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762957.868750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494098304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762957.868791:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762957.868798:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762957.868802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762957.868825:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2962 00000100:00080000:2.0:1789762957.868832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762957.868838:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762957.868846:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762957.868861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789762957.868869:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494098432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762957.868918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762957/real 1789762957] req@ffff8eb650f81180 x1876699494098048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762957.868937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494098048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762957.868954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762957.868958:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762957.868962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494098048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762957.868969:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762957.868992:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762957.869003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762957/real 1789762957] req@ffff8eb650f83b80 x1876699494098176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762957.869015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494098176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762957.869022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762957.869026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762957.869029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762957.869052:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762957.869067:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762957.869076:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762957/real 1789762957] req@ffff8eb650f81f80 x1876699494098304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762957.869087:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494098304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762957.869094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762957.869097:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762957.869100:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762957.869106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494098432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762957.869118:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762957.869126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762957/real 1789762957] req@ffff8eb650f83800 x1876699494098432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762957.869155:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494098432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762957.869168:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762957.869173:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762957.869179:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494098432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762958.572547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762959.596579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762960.620554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762961.644579:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762962.668564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762962.988566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762962.988576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762962.988580:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762962.988593:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2967 00000100:00080000:1.0:1789762962.988605:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762962.988613:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762962.988656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494098560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762962.988684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762962.988688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762962.988690:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762962.988695:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2967 00000100:00080000:1.0:1789762962.988699:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762962.988704:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762962.988716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762962.988746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494098560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762962.988795:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494098688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762962.988810:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762962.988815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762962.988817:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762962.988824:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762962.988824:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2967 00000100:00080000:1.0:1789762962.988833:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762962.988839:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762962.988847:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762962.988863:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494098816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762962.988879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762962/real 1789762962] req@ffff8eb65016c700 x1876699494098560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762962.988883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762962.988893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762962.988897:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762962.988903:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2967 00000100:00100000:0.0:1789762962.988908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494098560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762962.988910:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762962.988920:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:0.0:1789762962.988922:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762962.988929:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000008:00100000:1.0:1789762962.988929:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762962.988933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494098560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762962.988949:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494098944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762962.988961:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762962.988981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762962.988991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762962/real 1789762962] req@ffff8eb65016ce00 x1876699494098688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762962.988999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494098688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762962.989004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762962.989006:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762962.989008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762962.989012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098816:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762962.989023:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762962.989032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762962/real 1789762962] req@ffff8eb65016e300 x1876699494098816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762962.989043:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494098816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762962.989051:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762962.989054:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762962.989057:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494098816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762962.989072:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494098944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762962.989092:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762962.989103:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762962/real 1789762962] req@ffff8eb65016c380 x1876699494098944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762962.989120:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494098944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762962.989128:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762962.989131:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762962.989134:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494098944:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762963.692587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762964.716607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762965.740619:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762966.764647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762967.788564:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762968.108580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762968.108595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762968.108602:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762968.108616:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2972 00000100:00080000:0.0:1789762968.108632:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762968.108643:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762968.108714:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494099072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762968.108748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762968.108752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762968.108755:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762968.108759:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2972 00000100:00080000:0.0:1789762968.108764:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762968.108768:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762968.108779:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762968.108794:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494099200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762968.108807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762968.108811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762968.108815:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762968.108822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494099072:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789762968.108823:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2972 00000100:00080000:0.0:1789762968.108831:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762968.108839:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762968.108847:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762968.108858:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494099328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762968.108869:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762968.108873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762968.108876:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762968.108880:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2972 00000100:00080000:0.0:1789762968.108884:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762968.108888:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762968.108893:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762968.108904:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494099456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762968.108957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762968.109033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff8eb64304ca80 x1876699494099072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.109083:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494099072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.109103:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.109110:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762968.109118:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494099072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762968.109156:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.109213:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762968.109231:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff8eb64304f100 x1876699494099200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.109254:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494099200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.109268:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.109274:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762968.109280:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762968.109289:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762968.109319:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762968.109335:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff8eb64304c700 x1876699494099328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.109356:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494099328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.109370:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.109376:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762968.109382:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762968.109392:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494099456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762968.109422:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762968.109441:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762968/real 1789762968] req@ffff8eb64304ea00 x1876699494099456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762968.109490:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494099456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762968.109515:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762968.109520:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762968.109526:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494099456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762968.812543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762969.836564:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762970.860504:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762971.884582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762972.908598:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762973.228612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762973.228622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762973.228627:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762973.228637:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2977 00000100:00080000:1.0:1789762973.228666:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762973.228674:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762973.228710:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494099584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762973.228738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762973.228742:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762973.228744:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762973.228747:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2977 00000100:00080000:1.0:1789762973.228750:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762973.228754:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762973.228763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762973.228773:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494099712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762973.228783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762973.228787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494099584:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762973.228788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762973.228793:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762973.228803:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2977 00000100:00080000:1.0:1789762973.228806:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762973.228808:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762973.228812:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762973.228823:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494099840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762973.228832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762973.228835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762973.228836:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762973.228840:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2977 00000100:00080000:1.0:1789762973.228852:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762973.228856:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762973.228860:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762973.228866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762973.228871:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494099968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762973.228912:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff8eb65016fb80 x1876699494099584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762973.228929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494099584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762973.228939:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762973.228943:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762973.228946:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494099584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762973.228967:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762973.228985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762973.228995:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff8eb65016d500 x1876699494099712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762973.229005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494099712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762973.229011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762973.229014:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762973.229016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762973.229021:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762973.229044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762973.229058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff8eb65016d880 x1876699494099840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762973.229075:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494099840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762973.229087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762973.229092:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762973.229096:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494099840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762973.229104:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494099968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762973.229126:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762973.229136:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762973/real 1789762973] req@ffff8eb65016ea00 x1876699494099968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762973.229146:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494099968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762973.229153:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762973.229156:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762973.229158:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494099968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762973.932593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762974.956608:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762975.980586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762977.004558:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762978.028635:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762978.348503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762978.348511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762978.348516:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762978.348528:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2982 00000100:00080000:1.0:1789762978.348540:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762978.348550:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762978.348590:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494100096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762978.348633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762978.348638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762978.348640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762978.348646:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2982 00000100:00080000:1.0:1789762978.348653:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762978.348659:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762978.348669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762978.348684:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494100096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762978.348685:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494100224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762978.348710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762978.348713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762978.348721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762978.348728:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2982 00000100:00080000:1.0:1789762978.348731:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762978.348735:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762978.348740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762978.348756:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494100352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762978.348768:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762978.348773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762978.348776:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762978.348785:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762978.348792:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2982 00000100:00080000:1.0:1789762978.348798:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762978.348803:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762978.348809:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762978.348823:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494100480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762978.348835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff8eb65016f800 x1876699494100096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762978.348860:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494100096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762978.348870:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762978.348875:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762978.348883:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494100096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762978.348910:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762978.348927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762978.348936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff8eb65016e680 x1876699494100224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762978.348944:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494100224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762978.348950:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762978.348953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762978.348955:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762978.348959:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100352:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762978.348970:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762978.348980:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff8eb65016ea00 x1876699494100352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762978.348992:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494100352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762978.349008:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762978.349012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762978.349016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762978.349023:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494100480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762978.349044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762978.349056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762978/real 1789762978] req@ffff8eb65016d880 x1876699494100480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762978.349069:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494100480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762978.349087:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762978.349091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762978.349095:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494100480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762979.052623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762980.076550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762981.100693:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762982.124607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762983.148567:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762983.468520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762983.468533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762983.468539:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762983.468553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2987 00000100:00080000:1.0:1789762983.468567:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762983.468600:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762983.468660:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494100608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762983.468710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762983.468717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762983.468721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762983.468730:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2987 00000100:00080000:1.0:1789762983.468738:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762983.468747:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762983.468765:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762983.468792:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494100608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762983.468802:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494100736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762983.468825:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762983.468831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762983.468834:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762983.468839:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2987 00000100:00080000:1.0:1789762983.468845:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762983.468851:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762983.468859:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762983.468875:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494100864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762983.468890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762983.468895:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762983.468897:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762983.468903:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2987 00000100:00080000:1.0:1789762983.468908:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762983.468916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789762983.468921:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762983.468931:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762983.468944:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494100992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762983.468999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff8eb65016d500 x1876699494100608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762983.469035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494100608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762983.469060:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762983.469068:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762983.469076:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494100608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762983.469116:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762983.469163:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762983.469188:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff8eb65016fb80 x1876699494100736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762983.469204:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494100736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762983.469215:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762983.469219:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762983.469224:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762983.469232:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100864:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762983.469257:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762983.469269:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff8eb65016c380 x1876699494100864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762983.469283:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494100864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762983.469292:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762983.469296:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762983.469299:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494100864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762983.469307:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494100992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762983.469326:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762983.469337:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762983/real 1789762983] req@ffff8eb65016e300 x1876699494100992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762983.469351:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494100992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762983.469360:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762983.469363:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762983.469367:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494100992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762984.172650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762985.196550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762986.220520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762987.244519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762988.268556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762988.588500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762988.588510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762988.588513:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762988.588522:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2993 00000100:00080000:1.0:1789762988.588531:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762988.588551:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762988.588604:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494101120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762988.588632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762988.588634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762988.588636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762988.588639:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2993 00000100:00080000:1.0:1789762988.588642:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762988.588646:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762988.588654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762988.588664:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494101248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762988.588670:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494101120:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789762988.588675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762988.588679:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762988.588680:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762988.588683:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2993 00000100:00080000:1.0:1789762988.588686:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762988.588688:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762988.588692:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762988.588699:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494101376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762988.588706:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762988.588708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762988.588710:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762988.588713:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2993 00000100:00080000:1.0:1789762988.588718:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762988.588722:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789762988.588725:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762988.588733:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762988.588735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494101504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762988.588770:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff8eb65016ce00 x1876699494101120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762988.588784:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494101120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762988.588793:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762988.588796:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762988.588799:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494101120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762988.588817:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762988.588833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762988.588841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff8eb65016c700 x1876699494101248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762988.588849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494101248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762988.588863:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762988.588866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762988.588868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762988.588873:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762988.588886:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762988.588893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff8eb65016df80 x1876699494101376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762988.588904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494101376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762988.588913:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762988.588916:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762988.588920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762988.588926:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494101504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762988.588948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762988.588960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762988/real 1789762988] req@ffff8eb65016dc00 x1876699494101504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762988.588974:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494101504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762988.588980:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762988.588982:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762988.588984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494101504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762989.292551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762990.316595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762991.340608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762992.364645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762993.388616:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762993.708560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762993.708573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762993.708576:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762993.708584:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 2998 00000100:00080000:2.0:1789762993.708593:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762993.708599:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789762993.708631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494101632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762993.708657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762993.708662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762993.708663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762993.708666:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2998 00000100:00080000:2.0:1789762993.708668:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762993.708671:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762993.708678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762993.708686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494101760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762993.708693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762993.708697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762993.708698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762993.708700:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 2998 00000100:00080000:2.0:1789762993.708705:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762993.708708:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762993.708711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762993.708720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494101888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762993.708728:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762993.708734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762993.708735:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762993.708752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 2998 00000100:00100000:0.0:1789762993.708755:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494101632:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789762993.708755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762993.708769:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789762993.708775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762993.708788:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494102016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762993.708863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762993.708922:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff8eb650f80a80 x1876699494101632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762993.708938:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494101632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762993.708947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762993.708951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762993.708954:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494101632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789762993.708960:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762993.708980:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762993.708993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff8eb650f80000 x1876699494101760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762993.709014:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494101760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762993.709024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762993.709027:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762993.709031:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789762993.709060:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101888:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789762993.709078:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762993.709086:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff8eb650f80380 x1876699494101888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762993.709094:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494101888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762993.709100:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762993.709102:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762993.709104:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494101888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789762993.709108:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494102016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789762993.709118:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762993.709124:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762993/real 1789762993] req@ffff8eb650f81500 x1876699494102016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762993.709131:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494102016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762993.709141:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762993.709144:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789762993.709146:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494102016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789762994.412572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762995.436531:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762996.460621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762997.484548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762998.508548:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762998.828517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.828526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762998.828529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.828537:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3003 00000100:00080000:0.0:1789762998.828545:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.828551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789762998.828583:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494102144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.828607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.828610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.828611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.828614:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3003 00000100:00080000:0.0:1789762998.828616:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.828639:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762998.828648:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494102144:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789762998.828649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762998.828665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494102272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.828690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.828693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.828696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.828700:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3003 00000100:00080000:0.0:1789762998.828714:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.828718:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789762998.828722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762998.828724:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789762998.828736:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494102400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762998.828744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762998.828747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762998.828748:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762998.828752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3003 00000100:00080000:0.0:1789762998.828755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762998.828758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789762998.828762:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff8eb64304e680 x1876699494102144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789762998.828763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762998.828776:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494102528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762998.828783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494102144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762998.828793:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762998.828797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762998.828801:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494102144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789762998.828838:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762998.828864:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762998.828875:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff8eb64304f480 x1876699494102272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762998.828884:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494102272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762998.828890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762998.828892:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762998.828895:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789762998.828899:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789762998.828914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762998.828920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff8eb64304d880 x1876699494102400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762998.828928:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494102400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762998.828933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762998.828935:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762998.828938:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789762998.828941:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494102528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789762998.828951:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789762998.828956:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762998/real 1789762998] req@ffff8eb64304d500 x1876699494102528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762998.828964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494102528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762998.828968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762998.828970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789762998.828972:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494102528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789762999.532503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763000.556563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763001.580507:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763002.604508:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763003.628548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763003.948471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763003.948479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763003.948483:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763003.948491:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3008 00000100:00080000:0.0:1789763003.948500:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763003.948506:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763003.948550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494102656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763003.948570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763003.948573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763003.948575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763003.948577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3008 00000100:00080000:0.0:1789763003.948580:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763003.948582:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763003.948589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763003.948598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494102784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763003.948605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763003.948608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763003.948609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763003.948611:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3008 00000100:00080000:0.0:1789763003.948613:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763003.948618:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763003.948620:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494102656:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789763003.948621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763003.948631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494102912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763003.948644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763003.948647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763003.948648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763003.948650:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3008 00000100:00080000:0.0:1789763003.948653:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763003.948656:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763003.948659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763003.948666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494103040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763003.948708:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763003.948744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763003.948772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494103040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763004.652581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763005.676522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763006.700553:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763007.020661:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763007.020701:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763007.020718:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763007.020761:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763007.020773:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763007.020784:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763007.020795:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763003/real 1789763007] req@ffff8eb64304c380 x1876699494102912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763007.020820:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494102912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763007.020832:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763007.020837:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763007.020847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763007.020856:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763003/real 1789763007] req@ffff8eb64304f800 x1876699494103040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763007.020870:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494103040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763007.020878:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763007.020882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763007.020886:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494103040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763007.020893:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763003/real 1789763007] req@ffff8eb64304ce00 x1876699494102656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763007.020907:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494102656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763007.020918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763007.020920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763007.020923:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494102656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763007.020927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763003/real 1789763007] req@ffff8eb64304dc00 x1876699494102784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763007.020935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494102784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763007.020943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763007.020946:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763007.020949:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494102784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763007.724568:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763008.748557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763009.068508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.068519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763009.068523:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.068534:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3013 00000100:00080000:0.0:1789763009.068547:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.068554:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763009.068598:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494103168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.068628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.068631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.068633:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.068636:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3013 00000100:00080000:0.0:1789763009.068639:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.068643:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763009.068652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763009.068661:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494103296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.068674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.068677:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.068678:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.068682:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3013 00000100:00080000:0.0:1789763009.068685:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.068690:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763009.068696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763009.068706:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494103168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763009.068711:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494103424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763009.068726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763009.068730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763009.068731:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763009.068736:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3013 00000100:00080000:0.0:1789763009.068740:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763009.068744:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763009.068750:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763009.068764:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494103552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763009.068800:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763009.068854:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff8eb64304e300 x1876699494103168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763009.068877:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494103168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763009.068891:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763009.068895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763009.068899:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494103168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763009.068908:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763009.068938:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763009.068964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff8eb64304c000 x1876699494103296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763009.068980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494103296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763009.068989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763009.068993:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763009.068997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763009.069032:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763009.069058:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763009.069072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff8eb64304d180 x1876699494103424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763009.069086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494103424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763009.069095:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763009.069099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763009.069102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763009.069109:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494103552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763009.069132:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763009.069144:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763009/real 1789763009] req@ffff8eb64304ed80 x1876699494103552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763009.069165:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494103552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763009.069175:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763009.069185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763009.069189:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494103552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763009.772583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763010.796527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763011.820579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763012.844576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763013.868523:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763014.188503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763014.188515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763014.188519:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763014.188529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3018 00000100:00080000:0.0:1789763014.188540:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763014.188548:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763014.188587:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494103680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763014.188614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763014.188619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763014.188621:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763014.188626:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3018 00000100:00080000:0.0:1789763014.188633:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763014.188638:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763014.188649:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763014.188663:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494103808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763014.188666:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494103680:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763014.188684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763014.188688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763014.188691:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763014.188696:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3018 00000100:00080000:0.0:1789763014.188701:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763014.188705:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763014.188712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763014.188736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494103936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763014.188744:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763014.188753:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763014.188757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763014.188759:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763014.188765:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3018 00000100:00080000:0.0:1789763014.188768:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763014.188773:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763014.188778:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763014.188784:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff8eb64304ed80 x1876699494103680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763014.188792:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494104064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763014.188803:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494103680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763014.188813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763014.188817:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763014.188820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494103680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763014.188841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763014.188857:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763014.188865:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff8eb64304d180 x1876699494103808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763014.188873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494103808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763014.188883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763014.188885:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763014.188888:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763014.188891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763014.188901:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763014.188907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff8eb64304c000 x1876699494103936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763014.188915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494103936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763014.188923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763014.188926:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763014.188930:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494103936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763014.188936:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494104064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763014.188955:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763014.188966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763014/real 1789763014] req@ffff8eb64304e300 x1876699494104064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763014.188979:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494104064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763014.188984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763014.188986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763014.188989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494104064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763014.892552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763015.916663:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763016.940773:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763017.964756:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763018.988639:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763019.308489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763019.308499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763019.308503:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763019.308511:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3023 00000100:00080000:0.0:1789763019.308521:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763019.308528:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763019.308563:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494104192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763019.308584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763019.308587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763019.308589:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763019.308592:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3023 00000100:00080000:0.0:1789763019.308594:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763019.308597:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763019.308605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763019.308618:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494104320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763019.308631:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494104192:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763019.308632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763019.308641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763019.308643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763019.308649:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3023 00000100:00080000:0.0:1789763019.308653:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763019.308658:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763019.308665:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763019.308678:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494104448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763019.308686:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763019.308689:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763019.308691:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763019.308697:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763019.308698:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3023 00000100:00080000:0.0:1789763019.308705:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763019.308709:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763019.308713:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763019.308722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494104576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763019.308740:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff8eb64304dc00 x1876699494104192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763019.308760:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494104192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763019.308770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763019.308774:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763019.308778:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494104192:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763019.308798:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763019.308814:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763019.308842:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff8eb64304ce00 x1876699494104320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763019.308854:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494104320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763019.308862:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763019.308865:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763019.308868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763019.308874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104448:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763019.308890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763019.308899:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff8eb64304f800 x1876699494104448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763019.308910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494104448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763019.308918:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763019.308920:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763019.308923:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763019.308928:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494104576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763019.308942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763019.308951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763019/real 1789763019] req@ffff8eb64304c380 x1876699494104576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763019.308962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494104576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763019.308968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763019.308971:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763019.308973:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494104576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763020.012628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763021.036611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763022.060603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763023.084604:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763024.108554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763024.428572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.428585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763024.428590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.428603:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3028 00000100:00080000:0.0:1789763024.428617:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.428626:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763024.428674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494104704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.428714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.428719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.428721:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.428726:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3028 00000100:00080000:0.0:1789763024.428731:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.428735:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763024.428747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763024.428762:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494104832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.428776:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.428781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.428783:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.428788:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3028 00000100:00080000:0.0:1789763024.428794:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763024.428802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494104704:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763024.428805:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763024.428820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763024.428837:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494104960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763024.428851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763024.428856:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763024.428858:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763024.428864:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3028 00000100:00080000:0.0:1789763024.428882:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763024.428888:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763024.428894:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763024.428903:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763024.428913:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494105088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763024.428969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763024] req@ffff8eb64304d500 x1876699494104704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763024.428996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494104704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763024.429011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763024.429016:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763024.429022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494104704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763024.429031:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763024.429056:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763024.429071:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763024] req@ffff8eb64304d880 x1876699494104832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763024.429086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494104832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763024.429096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763024.429100:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763024.429104:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763024.429134:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763024.429155:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763024.429166:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763024] req@ffff8eb64304f480 x1876699494104960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763024.429181:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494104960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763024.429191:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763024.429194:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763024.429198:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494104960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763024.429206:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494105088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763024.429225:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763024.429236:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763024/real 1789763024] req@ffff8eb64304e680 x1876699494105088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763024.429250:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494105088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763024.429259:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763024.429262:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763024.429266:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494105088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763025.132604:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763026.156584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763027.180580:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763028.204671:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763029.228582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763029.548528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.548542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763029.548548:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.548561:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3034 00000100:00080000:0.0:1789763029.548575:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.548585:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763029.548631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494105216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.548669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.548675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.548677:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.548682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3034 00000100:00080000:0.0:1789763029.548686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.548691:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763029.548703:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763029.548720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494105344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.548737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763029.548739:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494105216:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763029.548744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.548750:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.548759:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3034 00000100:00080000:0.0:1789763029.548763:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.548768:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763029.548775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763029.548790:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494105472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763029.548802:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763029.548807:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763029.548810:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763029.548815:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3034 00000100:00080000:0.0:1789763029.548822:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763029.548827:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763029.548834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763029.548851:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494105600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763029.548877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763029.548950:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff8eb64304ea00 x1876699494105216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763029.548988:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494105216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763029.549003:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763029.549009:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763029.549015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494105216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763029.549045:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763029.549089:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763029.549105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff8eb64304c700 x1876699494105344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763029.549122:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494105344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763029.549132:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763029.549136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763029.549141:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763029.549148:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105472:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763029.549168:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763029.549180:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff8eb64304f100 x1876699494105472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763029.549194:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494105472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763029.549204:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763029.549207:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763029.549211:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763029.549218:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494105600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763029.549286:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763029.549311:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763029/real 1789763029] req@ffff8eb64304ca80 x1876699494105600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763029.549338:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494105600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763029.549369:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763029.549376:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763029.549383:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494105600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763030.252636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763031.276611:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763032.300625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763033.324611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763034.348607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763034.668526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763034.668536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763034.668551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763034.668560:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3039 00000100:00080000:2.0:1789763034.668571:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763034.668579:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763034.668619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494105728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763034.668648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763034.668652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763034.668653:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763034.668658:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3039 00000100:00080000:2.0:1789763034.668660:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763034.668669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763034.668678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763034.668690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494105856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763034.668700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763034.668703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763034.668705:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763034.668709:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3039 00000100:00080000:2.0:1789763034.668713:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763034.668719:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763034.668726:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494105728:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763034.668730:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763034.668752:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494105984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763034.668768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763034.668773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763034.668776:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763034.668783:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3039 00000100:00080000:2.0:1789763034.668789:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763034.668794:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763034.668799:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763034.668804:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763034.668821:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494106112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763034.668848:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff8eb650f81500 x1876699494105728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763034.668867:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494105728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763034.668877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763034.668881:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763034.668885:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494105728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763034.668891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763034.668908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763034.668918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff8eb650f80380 x1876699494105856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763034.668929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494105856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763034.668936:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763034.668938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763034.668941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763034.668983:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763034.669005:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763034.669018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff8eb650f80000 x1876699494105984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763034.669031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494105984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763034.669041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763034.669044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763034.669048:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494105984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763034.669054:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494106112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763034.669070:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763034.669080:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763034/real 1789763034] req@ffff8eb650f80a80 x1876699494106112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763034.669093:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494106112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763034.669102:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763034.669105:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763034.669108:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494106112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763035.372526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763036.396535:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763037.420559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763038.444614:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763039.468623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763039.788479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763039.788490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763039.788494:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763039.788509:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3044 00000100:00080000:2.0:1789763039.788542:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763039.788558:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763039.788668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494106240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763039.788711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763039.788716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763039.788719:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763039.788726:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3044 00000100:00080000:2.0:1789763039.788731:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763039.788737:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763039.788752:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763039.788769:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494106368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763039.788782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763039.788787:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763039.788790:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763039.788795:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3044 00000100:00080000:2.0:1789763039.788799:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763039.788805:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763039.788812:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763039.788826:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494106496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763039.788861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494106240:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763039.788867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763039.788875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763039.788877:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763039.788882:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3044 00000100:00080000:2.0:1789763039.788886:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763039.788890:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763039.788896:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763039.788922:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494106624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763039.788990:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763039.789052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff8eb650f83800 x1876699494106240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763039.789080:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494106240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763039.789096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763039.789102:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763039.789108:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494106240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763039.789118:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763039.789144:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763039.789191:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff8eb650f81f80 x1876699494106368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763039.789222:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494106368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763039.789229:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763039.789231:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763039.789233:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763039.789276:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763039.789299:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763039.789311:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff8eb650f83b80 x1876699494106496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763039.789324:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494106496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763039.789335:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763039.789337:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763039.789339:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763039.789343:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494106624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763039.789358:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763039.789366:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763039/real 1789763039] req@ffff8eb650f81180 x1876699494106624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763039.789373:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494106624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763039.789379:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763039.789380:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763039.789382:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494106624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763040.492573:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763041.516574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763042.540646:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763043.564597:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763044.588526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763044.908463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763044.908471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763044.908475:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763044.908483:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3049 00000100:00080000:2.0:1789763044.908493:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763044.908499:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763044.908530:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494106752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763044.908551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763044.908553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763044.908555:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763044.908558:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3049 00000100:00080000:2.0:1789763044.908560:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763044.908563:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763044.908571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763044.908579:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494106880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763044.908592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763044.908596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763044.908598:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763044.908603:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494106752:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763044.908605:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3049 00000100:00080000:2.0:1789763044.908613:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763044.908619:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763044.908626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763044.908641:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494107008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763044.908652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763044.908655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763044.908657:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763044.908660:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3049 00000100:00080000:2.0:1789763044.908663:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763044.908666:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763044.908670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763044.908671:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763044.908684:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494107136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763044.908715:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763044/real 1789763044] req@ffff8eb650f82a00 x1876699494106752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763044.908737:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494106752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763044.908745:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763044.908748:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763044.908751:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494106752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763044.908772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763044.908788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763044.908796:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763044/real 1789763044] req@ffff8eb650f83480 x1876699494106880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763044.908804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494106880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763044.908809:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763044.908812:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763044.908814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494106880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763044.908817:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107008:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763044.908827:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763044.908833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763044/real 1789763044] req@ffff8eb650f80e00 x1876699494107008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763044.908840:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494107008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763044.908845:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763044.908847:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763044.908849:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763044.908853:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494107136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763044.908862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763044.908868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763044/real 1789763044] req@ffff8eb650f80700 x1876699494107136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763044.908876:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494107136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763044.908881:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763044.908882:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763044.908884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494107136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763045.612627:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763046.636510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763047.660607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763048.684662:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763049.708554:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763050.028496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763050.028508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763050.028514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763050.028524:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3054 00000100:00080000:2.0:1789763050.028537:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763050.028545:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763050.028591:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494107264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763050.028631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763050.028637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763050.028640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763050.028646:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3054 00000100:00080000:2.0:1789763050.028651:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763050.028655:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763050.028667:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763050.028685:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494107392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763050.028700:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494107264:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763050.028700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763050.028724:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763050.028726:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763050.028731:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3054 00000100:00080000:2.0:1789763050.028734:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763050.028738:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763050.028744:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763050.028758:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494107520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763050.028775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763050.028779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763050.028782:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763050.028786:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3054 00000100:00080000:2.0:1789763050.028789:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763050.028797:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763050.028798:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763050.028806:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763050.028825:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494107648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763050.028852:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff8eb650f82680 x1876699494107264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763050.028874:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494107264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763050.028890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763050.028896:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763050.028902:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494107264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763050.028933:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763050.028966:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763050.028980:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff8eb650f82d80 x1876699494107392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763050.028993:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494107392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763050.029009:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763050.029012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763050.029015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763050.029020:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763050.029039:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763050.029048:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff8eb650f81880 x1876699494107520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763050.029059:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494107520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763050.029066:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763050.029069:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763050.029072:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763050.029077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494107648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763050.029157:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763050.029182:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763050/real 1789763050] req@ffff8eb650f81c00 x1876699494107648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763050.029208:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494107648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763050.029221:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763050.029225:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763050.029230:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494107648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763050.732618:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763051.756624:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763052.780657:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763053.804605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763054.828653:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763055.148624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.148655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763055.148661:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.148674:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3059 00000100:00080000:0.0:1789763055.148688:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.148698:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763055.148760:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494107776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.148811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.148819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.148823:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.148831:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3059 00000100:00080000:0.0:1789763055.148838:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.148845:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763055.148858:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763055.148883:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494107904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.148899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.148904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.148907:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.148916:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3059 00000100:00080000:0.0:1789763055.148928:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763055.148930:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494107776:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763055.148941:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763055.148962:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763055.149004:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494108032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763055.149016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763055.149020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763055.149022:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763055.149028:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3059 00000100:00080000:0.0:1789763055.149032:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763055.149038:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763055.149068:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763055.149102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494108160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763055.149122:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763055.149216:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff8eb64304ca80 x1876699494107776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763055.149248:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494107776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763055.149266:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763055.149274:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763055.149281:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494107776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763055.149323:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763055.149354:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763055.149370:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff8eb64304f100 x1876699494107904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763055.149386:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494107904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763055.149457:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763055.149462:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763055.149467:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494107904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763055.149476:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763055.149504:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763055.149520:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff8eb64304c700 x1876699494108032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763055.149536:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494108032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763055.149547:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763055.149551:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763055.149555:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763055.149562:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494108160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763055.149583:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763055.149594:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763055/real 1789763055] req@ffff8eb64304ea00 x1876699494108160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763055.149609:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494108160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763055.149619:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763055.149622:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763055.149626:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494108160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763055.852522:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763056.876518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763057.900522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763058.924646:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763059.948664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763060.268472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.268481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763060.268485:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.268495:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3064 00000100:00080000:0.0:1789763060.268505:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.268512:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763060.268545:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494108288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.268571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.268577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.268578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.268582:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3064 00000100:00080000:0.0:1789763060.268585:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.268587:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763060.268595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.268604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494108416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.268614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.268617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.268618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.268622:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3064 00000100:00080000:0.0:1789763060.268624:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.268627:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763060.268631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763060.268635:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494108288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763060.268643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494108544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763060.268653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763060.268655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763060.268657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763060.268661:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3064 00000100:00080000:0.0:1789763060.268663:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763060.268666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763060.268671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763060.268680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494108672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763060.268706:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763060.268741:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff8eb64304e680 x1876699494108288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763060.268758:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494108288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763060.268767:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763060.268771:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763060.268774:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494108288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763060.268780:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763060.268797:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763060.268806:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff8eb64304f480 x1876699494108416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763060.268816:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494108416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763060.268822:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763060.268824:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763060.268827:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763060.268848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108544:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763060.268893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763060.268905:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff8eb64304d880 x1876699494108544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763060.268936:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494108544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763060.268947:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763060.268951:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763060.268955:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763060.268963:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494108672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763060.268985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763060.268998:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763060/real 1789763060] req@ffff8eb64304d500 x1876699494108672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763060.269038:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494108672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763060.269049:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763060.269053:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763060.269074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494108672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763060.972640:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763061.996566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763063.020659:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763064.044613:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763065.068583:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763065.388491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.388502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763065.388506:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.388518:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3069 00000100:00080000:0.0:1789763065.388529:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.388536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763065.388574:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494108800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.388603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.388607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.388608:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.388612:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3069 00000100:00080000:0.0:1789763065.388615:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.388619:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763065.388628:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763065.388641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494108928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.388652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.388656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.388658:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.388661:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3069 00000100:00080000:0.0:1789763065.388665:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.388669:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763065.388674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763065.388679:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494108800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763065.388688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494109056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763065.388710:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763065.388714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763065.388716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763065.388720:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3069 00000100:00080000:0.0:1789763065.388724:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763065.388727:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763065.388732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763065.388743:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494109184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763065.388769:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763065.388839:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff8eb64304c380 x1876699494108800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763065.388869:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494108800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763065.388885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763065.388891:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763065.388897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494108800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763065.388907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763065.388939:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763065.388958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff8eb64304f800 x1876699494108928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763065.389000:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494108928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763065.389013:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763065.389018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763065.389023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494108928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763065.389054:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763065.389086:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763065.389104:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff8eb64304ce00 x1876699494109056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763065.389121:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494109056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763065.389132:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763065.389137:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763065.389142:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763065.389150:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494109184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763065.389177:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763065.389194:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763065/real 1789763065] req@ffff8eb64304dc00 x1876699494109184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763065.389211:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494109184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763065.389223:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763065.389228:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763065.389233:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494109184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763066.092664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763067.116649:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763068.140666:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763069.164616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763070.188697:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763070.508613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763070.508629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763070.508636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763070.508653:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3075 00000100:00080000:2.0:1789763070.508671:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763070.508683:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763070.508750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494109312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763070.508794:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763070.508799:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763070.508802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763070.508807:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3075 00000100:00080000:2.0:1789763070.508812:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763070.508817:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763070.508834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763070.508851:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494109440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763070.508868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763070.508872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763070.508875:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763070.508879:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3075 00000100:00080000:2.0:1789763070.508884:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763070.508890:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494109312:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763070.508895:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763070.508912:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763070.508927:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494109568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763070.508967:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763070.508973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763070.508977:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763070.508984:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3075 00000100:00080000:2.0:1789763070.508990:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763070.508996:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763070.509002:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763070.509020:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494109696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763070.509100:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763070.509143:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763070.509171:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494109696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763071.212692:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763072.236722:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763073.260629:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763073.580801:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789763073.580882:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763073.580932:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763073.581006:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763073.581025:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763073.581042:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763073.581109:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763073] req@ffff8eb650f81880 x1876699494109440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.581161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494109440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.581184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.581193:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763073.581201:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763073.581217:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763073] req@ffff8eb650f82d80 x1876699494109568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.581243:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494109568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.581261:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.581269:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763073.581276:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763073.581290:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763073] req@ffff8eb650f82680 x1876699494109696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.581315:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494109696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.581331:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.581339:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763073.581346:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494109696:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763073.581358:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763070/real 1789763073] req@ffff8eb650f81c00 x1876699494109312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.581383:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494109312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.581453:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.581461:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763073.581468:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494109312:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789763074.284669:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763075.308617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763075.628563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763075.628574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763075.628578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763075.628589:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3080 00000100:00080000:0.0:1789763075.628599:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763075.628623:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763075.628665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494109824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763075.628709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763075.628714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763075.628715:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763075.628718:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3080 00000100:00080000:0.0:1789763075.628721:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763075.628724:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763075.628733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763075.628753:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494109952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763075.628765:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763075.628769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763075.628771:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763075.628776:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3080 00000100:00080000:0.0:1789763075.628780:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763075.628785:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763075.628789:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494109824:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789763075.628793:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763075.628810:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494110080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763075.628830:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763075.628833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763075.628835:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763075.628841:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3080 00000100:00080000:0.0:1789763075.628848:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763075.628853:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763075.628860:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763075.628866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763075.628876:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494110208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763075.628927:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff8eb64304e300 x1876699494109824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763075.628950:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494109824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763075.628962:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763075.628967:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763075.628971:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494109824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763075.628979:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763075.629009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763075.629023:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff8eb64304c000 x1876699494109952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763075.629036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494109952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763075.629045:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763075.629048:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763075.629051:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494109952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763075.629077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763075.629099:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763075.629111:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff8eb64304d180 x1876699494110080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763075.629123:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494110080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763075.629137:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763075.629140:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763075.629143:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763075.629149:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494110208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763075.629176:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763075.629188:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763075/real 1789763075] req@ffff8eb64304ed80 x1876699494110208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763075.629201:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494110208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763075.629210:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763075.629214:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763075.629217:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494110208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763076.332612:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763077.356555:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763078.380515:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763079.404644:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763080.428636:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763080.748592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763080.748612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763080.748623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763080.748640:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3085 00000100:00080000:2.0:1789763080.748658:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763080.748671:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763080.748723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494110336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763080.748766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763080.748772:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763080.748774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763080.748780:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3085 00000100:00080000:2.0:1789763080.748785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763080.748791:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763080.748803:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763080.748819:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494110464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763080.748834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763080.748839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763080.748842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763080.748848:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3085 00000100:00080000:2.0:1789763080.748853:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763080.748881:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763080.748895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494110336:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763080.748897:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763080.748938:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494110592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763080.748968:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763080.748974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763080.748977:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763080.748983:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3085 00000100:00080000:2.0:1789763080.748992:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763080.748996:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763080.749002:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763080.749021:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494110720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763080.749039:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763080.749107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff8eb650f80700 x1876699494110336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763080.749135:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494110336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763080.749152:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763080.749159:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763080.749165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494110336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763080.749177:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763080.749205:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763080.749221:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff8eb650f80e00 x1876699494110464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763080.749236:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494110464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763080.749247:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763080.749252:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763080.749256:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110464:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763080.749294:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110592:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763080.749316:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763080.749329:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff8eb650f83480 x1876699494110592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763080.749343:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494110592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763080.749354:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763080.749357:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763080.749362:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110592:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763080.749369:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494110720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763080.749464:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763080.749489:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763080/real 1789763080] req@ffff8eb650f82a00 x1876699494110720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763080.749511:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494110720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763080.749522:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763080.749527:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763080.749531:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494110720:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763081.452506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763082.476586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763083.500635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763084.524658:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763085.548605:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763085.868528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763085.868542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763085.868548:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763085.868561:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3090 00000100:00080000:0.0:1789763085.868574:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763085.868584:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763085.868633:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494110848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763085.868671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763085.868677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763085.868679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763085.868685:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3090 00000100:00080000:0.0:1789763085.868689:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763085.868694:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763085.868706:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763085.868723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494110976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763085.868739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763085.868748:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763085.868749:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494110848:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763085.868751:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763085.868762:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3090 00000100:00080000:0.0:1789763085.868767:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763085.868777:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763085.868785:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763085.868801:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494111104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763085.868815:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763085.868840:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763085.868843:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763085.868849:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3090 00000100:00080000:0.0:1789763085.868855:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763085.868860:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763085.868867:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763085.868882:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494111232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763085.868913:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763085.868982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763085/real 1789763085] req@ffff8eb64304ed80 x1876699494110848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763085.869011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494110848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763085.869026:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763085.869031:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763085.869036:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494110848:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763085.869068:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763085.869102:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763085.869135:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763085/real 1789763085] req@ffff8eb64304d180 x1876699494110976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763085.869152:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494110976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763085.869163:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763085.869168:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763085.869172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494110976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763085.869180:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111104:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763085.869210:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763085.869226:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763085/real 1789763085] req@ffff8eb64304c000 x1876699494111104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763085.869241:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494111104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763085.869251:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763085.869255:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763085.869258:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763085.869266:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494111232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763085.869292:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763085.869303:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763085/real 1789763085] req@ffff8eb64304e300 x1876699494111232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763085.869317:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494111232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763085.869336:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763085.869340:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763085.869344:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494111232:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763086.572587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763087.596642:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763088.620592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763089.644633:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763090.668614:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763090.988516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763090.988529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763090.988536:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763090.988549:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3095 00000100:00080000:0.0:1789763090.988563:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763090.988572:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763090.988618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494111360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763090.988653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763090.988663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763090.988666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763090.988673:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3095 00000100:00080000:0.0:1789763090.988679:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763090.988685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763090.988698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763090.988713:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494111488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763090.988729:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763090.988734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763090.988740:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763090.988742:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494111360:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763090.988748:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3095 00000100:00080000:0.0:1789763090.988760:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763090.988765:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763090.988802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763090.988819:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494111616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763090.988833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763090.988838:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763090.988841:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763090.988846:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3095 00000100:00080000:0.0:1789763090.988851:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763090.988856:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763090.988864:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763090.988882:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763090.988883:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494111744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763090.988964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763090/real 1789763090] req@ffff8eb64304dc00 x1876699494111360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763090.988997:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494111360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763090.989012:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763090.989018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763090.989023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494111360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763090.989052:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763090.989103:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763090.989129:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763090/real 1789763090] req@ffff8eb64304ce00 x1876699494111488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763090.989155:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494111488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763090.989174:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763090.989181:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763090.989189:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111488:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763090.989202:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111616:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763090.989235:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763090.989251:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763090/real 1789763090] req@ffff8eb64304f800 x1876699494111616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763090.989266:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494111616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763090.989277:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763090.989280:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763090.989285:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494111616:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763090.989292:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494111744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763090.989328:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763090.989341:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763090/real 1789763090] req@ffff8eb64304c380 x1876699494111744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763090.989356:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494111744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763090.989366:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763090.989370:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763090.989374:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494111744:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763091.692572:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763092.716566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763093.740602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763094.764626:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763095.788741:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763096.108572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.108591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763096.108597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.108614:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3100 00000100:00080000:2.0:1789763096.108630:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.108641:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763096.108704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494111872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.108740:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.108769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763096.108772:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.108778:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3100 00000100:00080000:2.0:1789763096.108785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.108791:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763096.108805:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763096.108820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494111872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763096.108828:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494112000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.108846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.108851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763096.108853:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763096.108858:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3100 00000100:00080000:2.0:1789763096.108863:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.108867:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763096.108874:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763096.108890:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494112128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763096.108902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763096.108907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763096.108910:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763096.108918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763096.108924:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3100 00000100:00080000:2.0:1789763096.108933:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763096.108938:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763096.108946:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763096.108961:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494112256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763096.108982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff8eb650f81180 x1876699494111872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763096.109010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494111872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763096.109024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763096.109029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763096.109034:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494111872:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763096.109064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763096.109090:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763096.109105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff8eb650f83b80 x1876699494112000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763096.109121:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494112000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763096.109131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763096.109135:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763096.109139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763096.109146:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112128:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763096.109165:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763096.109175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff8eb650f81f80 x1876699494112128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763096.109198:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494112128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763096.109208:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763096.109211:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763096.109215:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763096.109221:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494112256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763096.109239:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763096.109250:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763096/real 1789763096] req@ffff8eb650f83800 x1876699494112256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763096.109264:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494112256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763096.109273:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763096.109276:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763096.109280:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494112256:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763096.812653:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763097.836663:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763098.860538:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763099.884600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763100.908643:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763101.228536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763101.228550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763101.228557:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763101.228571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3105 00000100:00080000:2.0:1789763101.228585:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763101.228594:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763101.228647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494112384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763101.228700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763101.228734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763101.228738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763101.228747:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3105 00000100:00080000:2.0:1789763101.228756:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763101.228764:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763101.228779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763101.228806:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494112512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763101.228823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763101.228828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763101.228834:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763101.228837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494112384:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763101.228843:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3105 00000100:00080000:2.0:1789763101.228856:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763101.228862:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763101.228869:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763101.228886:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494112640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763101.228922:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763101.228927:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763101.228930:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763101.228937:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3105 00000800:00000100:1.0:1789763101.228949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763101.228950:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763101.228963:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763101.228970:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763101.228987:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494112768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763101.229048:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff8eb650f80a80 x1876699494112384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763101.229085:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494112384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763101.229101:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763101.229108:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763101.229114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494112384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763101.229149:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763101.229183:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763101.229216:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff8eb650f80000 x1876699494112512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763101.229233:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494112512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763101.229243:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763101.229248:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763101.229252:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763101.229259:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112640:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763101.229282:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763101.229294:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff8eb650f80380 x1876699494112640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763101.229308:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494112640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763101.229318:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763101.229321:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763101.229325:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494112640:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763101.229333:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494112768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763101.229351:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763101.229362:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763101/real 1789763101] req@ffff8eb650f81500 x1876699494112768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763101.229376:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494112768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763101.229385:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763101.229389:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763101.229438:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494112768:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763101.932649:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763102.956551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763103.980671:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763105.004637:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763106.028555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763106.348512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763106.348537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763106.348543:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763106.348556:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3110 00000100:00080000:2.0:1789763106.348570:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763106.348579:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763106.348629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494112896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763106.348691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763106.348705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763106.348708:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763106.348721:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3110 00000100:00080000:2.0:1789763106.348728:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763106.348734:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763106.348749:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763106.348764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494112896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763106.348774:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494113024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763106.348792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763106.348797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763106.348800:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763106.348805:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3110 00000100:00080000:2.0:1789763106.348809:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763106.348814:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763106.348821:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763106.348835:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494113152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763106.348850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763106.348855:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763106.348861:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763106.348863:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763106.348874:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3110 00000100:00080000:2.0:1789763106.348882:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763106.348888:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763106.348896:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763106.348912:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494113280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763106.348931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff8eb650f81500 x1876699494112896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763106.348963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494112896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763106.348978:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763106.348984:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763106.348989:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494112896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763106.349017:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763106.349044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763106.349060:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff8eb650f80380 x1876699494113024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763106.349086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494113024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763106.349097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763106.349101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763106.349105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113024:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763106.349113:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113152:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763106.349133:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763106.349144:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff8eb650f80000 x1876699494113152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763106.349159:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494113152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763106.349169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763106.349173:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763106.349177:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113152:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763106.349198:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494113280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763106.349219:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763106.349232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763106/real 1789763106] req@ffff8eb650f80a80 x1876699494113280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763106.349246:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494113280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763106.349256:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763106.349260:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763106.349264:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494113280:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763107.052614:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763108.076615:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763109.100556:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763110.124619:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763111.148630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763111.468555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763111.468569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763111.468575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763111.468603:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3115 00000100:00080000:2.0:1789763111.468613:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763111.468620:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763111.468704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494113408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763111.468756:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763111.468760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763111.468762:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763111.468767:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3115 00000100:00080000:2.0:1789763111.468771:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763111.468774:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763111.468784:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763111.468797:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494113536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763111.468808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763111.468812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494113408:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763111.468813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763111.468817:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763111.468824:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3115 00000100:00080000:2.0:1789763111.468828:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763111.468830:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763111.468834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763111.468844:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494113664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763111.468859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763111.468862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763111.468863:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763111.468866:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3115 00000100:00080000:2.0:1789763111.468870:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763111.468885:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763111.468888:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763111.468894:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763111.468899:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494113792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763111.468943:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff8eb650f83800 x1876699494113408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763111.468962:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494113408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763111.468971:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763111.468974:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763111.468978:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494113408:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763111.468998:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763111.469034:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763111.469044:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff8eb650f81f80 x1876699494113536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763111.469053:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494113536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763111.469059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763111.469062:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763111.469065:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113536:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763111.469069:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113664:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763111.469082:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763111.469088:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff8eb650f83b80 x1876699494113664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763111.469100:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494113664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763111.469117:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763111.469121:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763111.469126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494113664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763111.469132:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494113792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763111.469173:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763111.469185:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763111/real 1789763111] req@ffff8eb650f81180 x1876699494113792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763111.469194:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494113792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763111.469201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763111.469203:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763111.469206:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494113792:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763112.172575:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763113.196571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763114.220569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763115.244666:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763116.268571:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763116.588535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763116.588548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763116.588553:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763116.588567:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3121 00000100:00080000:2.0:1789763116.588580:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763116.588589:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763116.588668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494113920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763116.588708:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763116.588713:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763116.588716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763116.588721:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3121 00000100:00080000:2.0:1789763116.588726:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763116.588731:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763116.588743:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763116.588758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494114048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763116.588773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763116.588778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763116.588782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763116.588786:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3121 00000100:00080000:2.0:1789763116.588790:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763116.588795:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763116.588801:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763116.588816:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494113920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763116.588817:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494114176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763116.588844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763116.588847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763116.588850:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763116.588856:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3121 00000100:00080000:2.0:1789763116.588860:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763116.588864:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763116.588870:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763116.588885:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494114304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763116.588916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763116.588971:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff8eb650f82a00 x1876699494113920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763116.588995:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494113920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763116.589010:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763116.589015:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763116.589020:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494113920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763116.589029:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763116.589055:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763116.589069:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff8eb650f83480 x1876699494114048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763116.589085:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494114048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763116.589094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763116.589098:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763116.589102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114048:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763116.589131:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114176:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763116.589151:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763116.589163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff8eb650f80e00 x1876699494114176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763116.589179:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494114176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763116.589188:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763116.589192:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763116.589195:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114176:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763116.589202:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494114304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763116.589220:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763116.589231:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763116/real 1789763116] req@ffff8eb650f80700 x1876699494114304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763116.589245:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494114304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763116.589255:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763116.589258:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763116.589262:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494114304:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763117.292659:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763118.316595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763119.340644:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763120.364606:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763121.388539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763121.708586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763121.708599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763121.708605:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763121.708619:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3126 00000100:00080000:0.0:1789763121.708632:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763121.708642:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763121.708689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494114432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763121.708724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763121.708728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763121.708731:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763121.708736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3126 00000100:00080000:0.0:1789763121.708740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763121.708745:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763121.708757:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763121.708775:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494114560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763121.708790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763121.708795:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763121.708802:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763121.708804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494114432:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763121.708812:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3126 00000100:00080000:0.0:1789763121.708820:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763121.708824:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763121.708830:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763121.708848:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494114688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763121.708861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763121.708866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763121.708869:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763121.708874:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3126 00000100:00080000:0.0:1789763121.708881:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763121.708885:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763121.708893:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763121.708902:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763121.708918:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494114816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763121.708964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff8eb64304d500 x1876699494114432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763121.708991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494114432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763121.709007:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763121.709012:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763121.709018:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494114432:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763121.709047:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763121.709072:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763121.709087:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff8eb64304d880 x1876699494114560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763121.709102:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494114560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763121.709113:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763121.709117:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763121.709121:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114560:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763121.709141:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114688:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763121.709164:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763121.709176:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff8eb64304f480 x1876699494114688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763121.709191:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494114688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763121.709201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763121.709205:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763121.709209:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494114688:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763121.709216:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494114816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763121.709235:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763121.709246:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763121/real 1789763121] req@ffff8eb64304e680 x1876699494114816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763121.709260:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494114816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763121.709269:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763121.709272:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763121.709276:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494114816:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763122.412618:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763123.436594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763124.460615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763125.484545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763126.508634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763126.828567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.828580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763126.828585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.828599:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3131 00000100:00080000:0.0:1789763126.828612:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.828621:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763126.828671:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494114944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.828709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.828718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.828720:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.828725:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3131 00000100:00080000:0.0:1789763126.828730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.828734:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763126.828747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763126.828764:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494115072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.828781:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763126.828789:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494114944:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763126.828790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.828797:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.828812:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3131 00000100:00080000:0.0:1789763126.828816:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.828820:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763126.828828:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763126.828844:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494115200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763126.828857:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763126.828862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763126.828865:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763126.828870:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3131 00000100:00080000:0.0:1789763126.828877:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763126.828881:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763126.828889:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763126.828896:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763126.828912:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494115328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763126.828955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff8eb64304ea00 x1876699494114944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763126.828983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494114944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763126.829013:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763126.829019:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763126.829024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494114944:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763126.829055:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763126.829089:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763126.829105:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff8eb64304c700 x1876699494115072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763126.829121:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494115072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763126.829131:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763126.829136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763126.829140:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763126.829148:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115200:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763126.829183:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763126.829205:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff8eb64304f100 x1876699494115200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763126.829231:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494115200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763126.829250:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763126.829257:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763126.829263:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763126.829275:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494115328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763126.829305:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763126.829334:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763126/real 1789763126] req@ffff8eb64304ca80 x1876699494115328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763126.829360:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494115328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763126.829371:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763126.829375:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763126.829384:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494115328:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763127.532668:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763128.556552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763129.580630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763130.604663:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763131.628632:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763131.948635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763131.948649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763131.948655:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763131.948668:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3136 00000100:00080000:0.0:1789763131.948683:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763131.948693:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763131.948743:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494115456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763131.948778:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763131.948782:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763131.948785:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763131.948790:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3136 00000100:00080000:0.0:1789763131.948795:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763131.948800:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763131.948814:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763131.948834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494115456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763131.948835:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494115584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763131.948862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763131.948866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763131.948869:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763131.948874:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3136 00000100:00080000:0.0:1789763131.948880:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763131.948885:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763131.948902:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763131.948918:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494115712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763131.948932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763131.948937:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763131.948939:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763131.948948:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763131.948949:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3136 00000100:00080000:0.0:1789763131.948961:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763131.948976:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763131.948983:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763131.948998:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494115840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763131.949025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763131/real 1789763131] req@ffff8eb64304ca80 x1876699494115456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763131.949053:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494115456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763131.949067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763131.949072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763131.949077:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494115456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763131.949115:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763131.949143:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763131.949158:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763131/real 1789763131] req@ffff8eb64304f100 x1876699494115584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763131.949173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494115584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763131.949184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763131.949188:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763131.949193:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115584:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763131.949217:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115712:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763131.949240:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763131.949253:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763131/real 1789763131] req@ffff8eb64304c700 x1876699494115712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763131.949268:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494115712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763131.949278:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763131.949282:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763131.949286:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494115712:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763131.949294:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494115840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763131.949318:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763131.949329:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763131/real 1789763131] req@ffff8eb64304ea00 x1876699494115840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763131.949342:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494115840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763131.949351:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763131.949354:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763131.949359:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494115840:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763132.652596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763133.676576:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763134.700689:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763135.724597:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763136.748607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763137.068572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763137.068589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763137.068597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763137.068612:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3141 00000100:00080000:2.0:1789763137.068629:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763137.068640:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763137.068694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494115968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763137.068740:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763137.068747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763137.068750:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763137.068758:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3141 00000100:00080000:2.0:1789763137.068765:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763137.068772:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763137.068786:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763137.068814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494115968:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763137.068816:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494116096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763137.068844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763137.068850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763137.068854:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763137.068861:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3141 00000100:00080000:2.0:1789763137.068869:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763137.068876:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763137.068886:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763137.068908:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494116224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763137.068958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763137.068965:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116096:192.168.203.160@tcp:38:kworker.0 00000100:00080000:2.0:1789763137.068968:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763137.068977:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763137.068988:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3141 00000100:00080000:2.0:1789763137.068997:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763137.069005:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763137.069014:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116224:192.168.203.160@tcp:8:kworker.0 00000008:00100000:2.0:1789763137.069016:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763137.069065:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494116352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763137.069116:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494116352:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763137.772629:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763138.796605:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763139.820682:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763140.140709:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789763140.140764:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763140.140800:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763140.140848:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763140.140874:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763140.140881:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763140.140970:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763140] req@ffff8eb650f81880 x1876699494116352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763140.140998:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494116352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763140.141009:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763140.141015:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763140.141018:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494116352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763140.141025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763140] req@ffff8eb650f82680 x1876699494116096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763140.141035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494116096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763140.141042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763140.141044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763140.141047:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116096:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763140.141055:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763140] req@ffff8eb650f82d80 x1876699494116224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763140.141072:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494116224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763140.141082:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763140.141087:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763140.141092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116224:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763140.141101:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763137/real 1789763140] req@ffff8eb650f81c00 x1876699494115968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763140.141117:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494115968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763140.141128:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763140.141132:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763140.141136:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494115968:192.168.203.160@tcp:250:kworker.0 00010000:00010000:1.0:1789763140.844667:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763141.868572:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763142.188529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763142.188543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763142.188549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763142.188563:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3146 00000100:00080000:2.0:1789763142.188578:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763142.188588:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763142.188631:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494116480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763142.188667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763142.188672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763142.188675:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763142.188680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3146 00000100:00080000:2.0:1789763142.188687:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763142.188695:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763142.188712:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763142.188739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494116608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763142.188763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763142.188767:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494116480:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763142.188774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763142.188783:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763142.188795:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3146 00000100:00080000:2.0:1789763142.188804:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763142.188813:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763142.188826:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763142.188854:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494116736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763142.188876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763142.188885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789763142.188888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763142.188891:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763142.188906:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3146 00000100:00080000:2.0:1789763142.188916:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763142.188926:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763142.188938:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763142.188969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff8eb650f81c00 x1876699494116480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763142.188971:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494116864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763142.189030:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494116480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763142.189062:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763142.189070:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763142.189078:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494116480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763142.189148:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763142.189202:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763142.189232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff8eb650f82d80 x1876699494116608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763142.189262:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494116608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763142.189280:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763142.189289:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763142.189297:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763142.189321:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116736:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763142.189370:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763142.189480:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff8eb650f82680 x1876699494116736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763142.189511:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494116736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763142.189529:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763142.189536:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763142.189544:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494116736:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763142.189557:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494116864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763142.189603:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763142.189628:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763142/real 1789763142] req@ffff8eb650f81880 x1876699494116864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763142.189653:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494116864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763142.189681:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763142.189689:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763142.189696:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494116864:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763142.892618:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763143.916631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763144.940637:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763145.964626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763146.988655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763147.308547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763147.308557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763147.308560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763147.308571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3151 00000100:00080000:0.0:1789763147.308581:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763147.308589:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763147.308623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494116992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763147.308650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763147.308657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763147.308659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763147.308662:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3151 00000100:00080000:0.0:1789763147.308665:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763147.308669:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763147.308679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763147.308691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494117120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763147.308704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763147.308705:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494116992:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763147.308711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763147.308724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763147.308731:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3151 00000100:00080000:0.0:1789763147.308733:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763147.308736:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763147.308741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763147.308752:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494117248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763147.308760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763147.308764:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763147.308766:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763147.308770:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3151 00000100:00080000:0.0:1789763147.308797:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763147.308801:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763147.308806:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763147.308812:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763147.308818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494117376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763147.308857:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff8eb64304e680 x1876699494116992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763147.308875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494116992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763147.308885:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763147.308889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763147.308893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494116992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763147.308915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763147.308949:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763147.308959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff8eb64304f480 x1876699494117120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763147.308970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494117120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763147.308977:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763147.308981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763147.308983:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117120:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763147.308988:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117248:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763147.309002:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763147.309010:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff8eb64304d880 x1876699494117248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763147.309020:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494117248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763147.309036:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763147.309039:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763147.309042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117248:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763147.309047:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494117376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763147.309062:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763147.309071:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763147/real 1789763147] req@ffff8eb64304d500 x1876699494117376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763147.309088:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494117376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763147.309094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763147.309097:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763147.309099:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494117376:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763148.012556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763149.036570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763150.060623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763151.084655:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763152.108738:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763152.428623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763152.428639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763152.428649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763152.428669:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3156 00000100:00080000:2.0:1789763152.428687:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763152.428700:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763152.428762:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494117504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763152.428805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763152.428810:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763152.428813:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763152.428819:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3156 00000100:00080000:2.0:1789763152.428824:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763152.428837:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763152.428852:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763152.428870:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494117632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763152.428885:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763152.428890:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763152.428892:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763152.428896:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3156 00000100:00080000:2.0:1789763152.428899:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763152.428904:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763152.428912:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763152.428917:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494117504:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763152.428930:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494117760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763152.428967:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763152.428975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763152.428980:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763152.428989:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3156 00000100:00080000:2.0:1789763152.428997:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763152.429006:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763152.429017:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763152.429031:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763152.429047:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494117888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763152.429099:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff8eb650f80700 x1876699494117504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763152.429129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494117504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763152.429145:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763152.429150:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763152.429156:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494117504:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763152.429191:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763152.429229:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763152.429243:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff8eb650f80e00 x1876699494117632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763152.429259:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494117632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763152.429279:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763152.429283:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763152.429288:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763152.429294:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117760:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763152.429315:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763152.429350:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff8eb650f83480 x1876699494117760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763152.429376:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494117760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763152.429387:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763152.429432:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763152.429438:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494117760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763152.429445:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494117888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763152.429471:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763152.429486:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763152/real 1789763152] req@ffff8eb650f82a00 x1876699494117888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763152.429502:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494117888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763152.429512:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763152.429516:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763152.429521:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494117888:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763153.132637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763154.156635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763155.180704:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763156.204682:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763157.228669:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763157.548542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763157.548557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763157.548563:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763157.548579:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3162 00000100:00080000:2.0:1789763157.548596:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763157.548608:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763157.548680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494118016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763157.548722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763157.548727:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763157.548730:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763157.548736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3162 00000100:00080000:2.0:1789763157.548741:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763157.548747:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763157.548763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763157.548784:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494118144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763157.548800:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494118016:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763157.548805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763157.548826:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763157.548831:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763157.548838:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3162 00000100:00080000:2.0:1789763157.548843:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763157.548864:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763157.548876:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763157.548904:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494118272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763157.548916:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763157.548941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763157.548950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763157.548954:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763157.548963:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3162 00000100:00080000:2.0:1789763157.548971:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763157.548981:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763157.548987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763157] req@ffff8eb650f81180 x1876699494118016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763157.549000:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763157.549013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494118016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763157.549035:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494118400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763157.549040:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763157.549057:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763157.549077:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494118016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763157.549128:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763157.549161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763157.549174:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763157] req@ffff8eb650f83b80 x1876699494118144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763157.549185:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494118144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763157.549193:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763157.549196:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763157.549199:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118144:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763157.549205:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118272:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763157.549222:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763157.549230:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763157] req@ffff8eb650f81f80 x1876699494118272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763157.549239:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494118272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763157.549246:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763157.549248:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763157.549250:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118272:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763157.549256:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494118400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763157.549269:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763157.549276:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763157/real 1789763157] req@ffff8eb650f83800 x1876699494118400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763157.549285:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494118400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763157.549310:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763157.549313:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763157.549316:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494118400:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763158.252671:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763159.276633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763160.300631:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763161.324640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763162.348576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763162.668612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.668628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763162.668637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.668654:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3167 00000100:00080000:2.0:1789763162.668673:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.668685:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763162.668753:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494118528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.668799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.668805:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.668807:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.668813:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3167 00000100:00080000:2.0:1789763162.668817:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.668822:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763162.668837:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.668854:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494118656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763162.668868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.668874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.668877:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.668881:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3167 00000100:00080000:2.0:1789763162.668886:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.668891:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763162.668898:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763162.668916:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494118784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763162.668918:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494118528:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763162.668972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763162.668977:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763162.668981:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763162.668989:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3167 00000100:00080000:2.0:1789763162.668993:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763162.668999:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763162.669005:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763162.669022:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494118912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763162.669051:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763162.669110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff8eb650f80a80 x1876699494118528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763162.669137:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494118528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763162.669152:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763162.669158:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763162.669163:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494118528:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763162.669173:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763162.669198:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763162.669213:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff8eb650f80000 x1876699494118656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763162.669228:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494118656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763162.669238:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763162.669273:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763162.669278:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763162.669310:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118784:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763162.669336:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763162.669350:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff8eb650f80380 x1876699494118784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763162.669366:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494118784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763162.669385:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763162.669389:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763162.669436:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494118784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763162.669445:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494118912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763162.669470:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763162.669496:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763162/real 1789763162] req@ffff8eb650f81500 x1876699494118912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763162.669512:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494118912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763162.669523:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763162.669527:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763162.669530:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494118912:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763163.372638:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763164.396618:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763165.420711:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763166.444593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763167.468590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763167.788572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.788581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763167.788584:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.788593:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3172 00000100:00080000:0.0:1789763167.788601:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.788607:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763167.788640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494119040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.788664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.788666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.788668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.788670:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3172 00000100:00080000:0.0:1789763167.788673:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.788675:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763167.788683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763167.788690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494119168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.788697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.788700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.788701:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.788703:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3172 00000100:00080000:0.0:1789763167.788706:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.788708:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763167.788711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763167.788716:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494119296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763167.788723:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763167.788725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763167.788727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763167.788730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3172 00000100:00080000:0.0:1789763167.788732:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763167.788735:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763167.788738:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763167.788744:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494119424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763167.788756:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494119040:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789763167.788814:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763167.788848:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff8eb64304c380 x1876699494119040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763167.788863:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494119040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763167.788871:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763167.788874:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763167.788877:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494119040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763167.788882:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763167.788895:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763167.788902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff8eb64304f800 x1876699494119168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763167.788910:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494119168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763167.788915:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763167.788917:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763167.788920:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763167.788923:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119296:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763167.788933:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763167.788939:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff8eb64304ce00 x1876699494119296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763167.788946:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494119296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763167.788951:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763167.788953:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763167.788955:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119296:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763167.788975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494119424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763167.788985:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763167.788992:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763167/real 1789763167] req@ffff8eb64304dc00 x1876699494119424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763167.788999:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494119424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763167.789004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763167.789006:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763167.789007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494119424:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763168.492693:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763169.516639:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763170.540707:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763171.564602:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763172.588648:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763172.908620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763172.908639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763172.908646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763172.908663:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3177 00000100:00080000:2.0:1789763172.908679:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763172.908689:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763172.908751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494119552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763172.908799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763172.908804:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763172.908807:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763172.908813:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3177 00000100:00080000:2.0:1789763172.908818:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763172.908822:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763172.908836:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763172.908853:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494119680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763172.908872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763172.908874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494119552:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763172.908879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763172.908888:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763172.908896:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3177 00000100:00080000:2.0:1789763172.908900:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763172.908905:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763172.908912:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763172.908927:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494119808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763172.908941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763172.908946:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763172.908949:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763172.908954:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3177 00000100:00080000:2.0:1789763172.908958:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763172.908963:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763172.908970:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763172.908986:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494119936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763172.908995:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763172.909087:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763172/real 1789763172] req@ffff8eb650f81500 x1876699494119552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763172.909115:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494119552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763172.909140:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763172.909145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763172.909151:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494119552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763172.909208:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763172.909270:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763172.909289:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763172/real 1789763172] req@ffff8eb650f80380 x1876699494119680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763172.909316:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494119680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763172.909328:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763172.909332:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763172.909336:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119680:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763172.909344:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119808:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763172.909367:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763172.909379:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763172/real 1789763172] req@ffff8eb650f80000 x1876699494119808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763172.909434:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494119808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763172.909446:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763172.909450:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763172.909455:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494119808:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763172.909463:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494119936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763172.909491:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763172.909530:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763172/real 1789763172] req@ffff8eb650f80a80 x1876699494119936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763172.909556:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494119936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763172.909587:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763172.909594:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763172.909601:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494119936:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763173.612678:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763174.636596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763175.660623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763176.684572:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763177.708631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763178.028594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763178.028607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763178.028612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763178.028625:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3182 00000100:00080000:0.0:1789763178.028639:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763178.028648:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763178.028697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494120064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763178.028749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763178.028761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763178.028765:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763178.028773:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3182 00000100:00080000:0.0:1789763178.028782:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763178.028791:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763178.028808:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763178.028830:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494120192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763178.028845:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494120064:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763178.028851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763178.028865:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763178.028868:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763178.028874:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3182 00000100:00080000:0.0:1789763178.028879:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763178.028883:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763178.028890:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763178.028905:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494120320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763178.028919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763178.028924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763178.028926:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763178.028931:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3182 00000100:00080000:0.0:1789763178.028937:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763178.028942:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763178.028949:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763178.028958:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763178.028969:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494120448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763178.029021:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff8eb64304e300 x1876699494120064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763178.029048:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494120064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763178.029063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763178.029068:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763178.029074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494120064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763178.029104:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763178.029131:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763178.029294:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff8eb64304c000 x1876699494120192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763178.029312:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494120192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763178.029323:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763178.029328:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763178.029333:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763178.029341:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120320:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763178.029369:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763178.029382:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff8eb64304d180 x1876699494120320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763178.029442:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494120320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763178.029453:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763178.029457:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763178.029461:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763178.029469:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494120448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763178.029493:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763178.029507:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763178/real 1789763178] req@ffff8eb64304ed80 x1876699494120448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763178.029521:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494120448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763178.029531:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763178.029534:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763178.029537:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494120448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763178.732588:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763179.756605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763180.780639:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763181.804661:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763182.828640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763183.148577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.148608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763183.148617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.148634:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3187 00000100:00080000:2.0:1789763183.148651:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.148664:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763183.148720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494120576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.148760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.148765:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763183.148768:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.148774:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3187 00000100:00080000:2.0:1789763183.148779:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.148784:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763183.148796:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763183.148814:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494120704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.148829:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763183.148839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494120576:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763183.148840:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763183.148845:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.148859:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3187 00000100:00080000:2.0:1789763183.148864:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.148868:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763183.148875:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763183.148893:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494120832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763183.148916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763183.148921:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763183.148924:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763183.148929:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3187 00000100:00080000:2.0:1789763183.148934:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763183.148938:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763183.148942:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763183.148963:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494120960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763183.148964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763183.149030:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff8eb650f83800 x1876699494120576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763183.149058:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494120576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763183.149072:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763183.149077:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763183.149083:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494120576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763183.149138:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763183.149173:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763183.149191:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff8eb650f81f80 x1876699494120704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763183.149208:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494120704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763183.149219:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763183.149223:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763183.149228:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120704:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763183.149236:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120832:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763183.149257:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763183.149268:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff8eb650f83b80 x1876699494120832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763183.149293:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494120832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763183.149302:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763183.149306:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763183.149309:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494120832:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763183.149317:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494120960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763183.149336:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763183.149347:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763183/real 1789763183] req@ffff8eb650f81180 x1876699494120960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763183.149361:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494120960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763183.149371:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763183.149374:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763183.149377:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494120960:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763183.852586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763184.876670:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763185.900613:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763186.924589:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763187.948532:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763188.268527:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763188.268545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763188.268552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763188.268569:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3192 00000100:00080000:2.0:1789763188.268586:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763188.268599:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763188.268656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494121088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763188.268704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763188.268712:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763188.268715:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763188.268722:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3192 00000100:00080000:2.0:1789763188.268728:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763188.268734:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763188.268750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763188.268771:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494121216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763188.268790:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494121088:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763188.268790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763188.268807:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763188.268812:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763188.268818:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3192 00000100:00080000:2.0:1789763188.268821:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763188.268826:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763188.268833:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763188.268850:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494121344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763188.268864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763188.268869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763188.268872:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763188.268877:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3192 00000100:00080000:2.0:1789763188.268881:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763188.268888:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763188.268897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763188.268901:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763188.268921:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494121472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763188.268961:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff8eb650f82a00 x1876699494121088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763188.268989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494121088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763188.269005:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763188.269010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763188.269015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494121088:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763188.269044:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763188.269116:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763188.269144:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff8eb650f83480 x1876699494121216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763188.269172:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494121216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763188.269190:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763188.269197:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763188.269204:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121216:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763188.269215:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121344:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763188.269246:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763188.269261:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff8eb650f80e00 x1876699494121344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763188.269277:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494121344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763188.269301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763188.269310:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763188.269315:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121344:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763188.269323:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494121472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763188.269345:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763188.269360:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763188/real 1789763188] req@ffff8eb650f80700 x1876699494121472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763188.269384:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494121472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763188.269452:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763188.269460:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763188.269468:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494121472:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763188.972637:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763189.996654:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763191.020628:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763192.044687:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763193.068604:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763193.388509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763193.388522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763193.388528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763193.388543:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3197 00000100:00080000:2.0:1789763193.388557:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763193.388569:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763193.388614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494121600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763193.388650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763193.388656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763193.388660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763193.388665:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3197 00000100:00080000:2.0:1789763193.388669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763193.388673:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763193.388684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763193.388702:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494121728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763193.388716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763193.388721:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763193.388723:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763193.388728:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3197 00000100:00080000:2.0:1789763193.388732:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763193.388740:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494121600:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763193.388742:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763193.388757:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763193.388774:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494121856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763193.388797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763193.388801:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763193.388804:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763193.388810:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3197 00000100:00080000:2.0:1789763193.388816:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763193.388821:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763193.388826:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763193.388842:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763193.388844:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494121984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763193.388918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff8eb650f81880 x1876699494121600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763193.388949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494121600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763193.388964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763193.388970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763193.388976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494121600:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763193.388985:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763193.389017:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763193.389056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff8eb650f82680 x1876699494121728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763193.389073:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494121728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763193.389084:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763193.389088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763193.389092:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763193.389121:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121856:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763193.389146:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763193.389159:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff8eb650f82d80 x1876699494121856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763193.389173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494121856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763193.389184:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763193.389187:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763193.389191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494121856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763193.389197:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494121984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763193.389215:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763193.389226:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763193/real 1789763193] req@ffff8eb650f81c00 x1876699494121984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763193.389249:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494121984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763193.389258:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763193.389262:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763193.389265:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494121984:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763194.092578:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763195.116592:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763196.140604:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763197.164607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763198.188600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763198.508500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.508528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763198.508533:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.508547:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3203 00000100:00080000:2.0:1789763198.508560:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.508570:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763198.508618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494122112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.508653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.508658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.508660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.508666:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3203 00000100:00080000:2.0:1789763198.508669:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.508674:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763198.508687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763198.508701:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494122240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.508716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.508723:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.508727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.508736:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3203 00000100:00080000:2.0:1789763198.508743:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.508751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763198.508763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763198.508773:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494122112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763198.508795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494122368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763198.508822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763198.508831:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763198.508835:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763198.508842:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3203 00000100:00080000:2.0:1789763198.508850:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763198.508855:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763198.508865:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763198.508866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763198.508894:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494122496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763198.508947:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff8eb650f81c00 x1876699494122112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763198.508976:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494122112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763198.509017:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763198.509024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763198.509029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494122112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763198.509039:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763198.509067:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763198.509082:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff8eb650f82d80 x1876699494122240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763198.509098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494122240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763198.509117:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763198.509122:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763198.509126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122240:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763198.509155:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122368:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763198.509175:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763198.509187:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff8eb650f82680 x1876699494122368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763198.509201:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494122368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763198.509211:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763198.509214:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763198.509218:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122368:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763198.509225:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494122496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763198.509243:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763198.509254:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763198/real 1789763198] req@ffff8eb650f81880 x1876699494122496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763198.509267:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494122496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763198.509277:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763198.509280:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763198.509285:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494122496:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763199.212609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763200.236577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763201.260571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763202.284657:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763203.308543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763203.628553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763203.628567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763203.628572:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763203.628599:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3208 00000100:00080000:0.0:1789763203.628613:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763203.628622:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763203.628669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494122624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763203.628717:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763203.628725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763203.628729:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763203.628738:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3208 00000100:00080000:0.0:1789763203.628746:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763203.628755:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763203.628773:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763203.628793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494122624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763203.628804:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494122752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763203.628823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763203.628828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763203.628831:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763203.628836:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3208 00000100:00080000:0.0:1789763203.628844:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763203.628851:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763203.628859:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763203.628889:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494122880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763203.628904:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763203.628909:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763203.628911:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763203.628919:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3208 00000100:00080000:0.0:1789763203.628924:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763203.628930:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763203.628938:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763203.628976:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494123008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763203.628987:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763203.629040:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763203.629082:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494123008:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763204.332518:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763205.356615:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763206.380606:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763206.700805:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763206.700867:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763206.700896:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763206.700999:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763206.701013:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763206.701023:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763206.701119:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763206] req@ffff8eb64304e300 x1876699494123008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763206.701173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494123008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763206.701191:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763206.701198:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763206.701205:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494123008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763206.701216:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763206] req@ffff8eb64304d180 x1876699494122752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763206.701238:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494122752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763206.701254:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763206.701261:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763206.701270:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763206.701285:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763206] req@ffff8eb64304c000 x1876699494122880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763206.701311:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494122880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763206.701327:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763206.701335:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763206.701342:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494122880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763206.701359:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763203/real 1789763206] req@ffff8eb64304ed80 x1876699494122624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763206.701471:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494122624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763206.701491:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763206.701501:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763206.701508:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494122624:192.168.203.160@tcp:250:kworker.0 00010000:00010000:3.0:1789763207.404657:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763208.428620:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763208.748614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763208.748631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763208.748636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763208.748649:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3213 00000100:00080000:2.0:1789763208.748662:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763208.748672:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763208.748727:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494123136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763208.748785:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763208.748792:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763208.748796:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763208.748805:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3213 00000100:00080000:2.0:1789763208.748811:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763208.748817:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763208.748830:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763208.748852:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494123264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763208.748870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763208.748876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763208.748881:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763208.748884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494123136:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763208.748889:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3213 00000100:00080000:2.0:1789763208.748898:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763208.748909:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763208.748946:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763208.748974:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494123392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763208.749006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763208.749031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763208.749035:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763208.749042:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3213 00000100:00080000:2.0:1789763208.749050:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763208.749052:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763208.749062:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763208.749078:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763208.749103:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494123520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763208.749168:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff8eb650f80700 x1876699494123136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763208.749206:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494123136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763208.749236:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763208.749243:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763208.749251:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494123136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763208.749302:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763208.749348:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763208.749373:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff8eb650f80e00 x1876699494123264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763208.749449:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494123264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763208.749483:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763208.749488:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763208.749493:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763208.749503:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123392:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763208.749539:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763208.749567:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff8eb650f83480 x1876699494123392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763208.749590:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494123392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763208.749603:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763208.749609:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763208.749614:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123392:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763208.749625:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494123520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763208.749659:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763208.749678:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763208/real 1789763208] req@ffff8eb650f82a00 x1876699494123520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763208.749698:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494123520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763208.749709:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763208.749712:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763208.749715:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494123520:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763209.452661:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763210.476613:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763211.500596:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763212.524635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763213.548657:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763213.868488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763213.868502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763213.868508:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763213.868523:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3218 00000100:00080000:2.0:1789763213.868538:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763213.868549:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763213.868631:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494123648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763213.868690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763213.868696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763213.868699:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763213.868705:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3218 00000100:00080000:2.0:1789763213.868718:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763213.868725:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763213.868740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763213.868766:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494123776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763213.868779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763213.868784:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763213.868787:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763213.868792:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3218 00000100:00080000:2.0:1789763213.868796:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763213.868800:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763213.868805:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763213.868820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494123648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763213.868821:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494123904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763213.868976:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763213.869005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763213.869012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763213.869015:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763213.869022:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3218 00000100:00080000:2.0:1789763213.869030:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763213.869039:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763213.869048:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763213.869055:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763213/real 1789763213] req@ffff8eb650f81180 x1876699494123648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763213.869072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494124032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763213.869090:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494123648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763213.869108:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763213.869116:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763213.869123:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494123648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763213.869134:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763213.869163:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763213.869180:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763213/real 1789763213] req@ffff8eb650f83b80 x1876699494123776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763213.869195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494123776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763213.869206:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763213.869210:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763213.869215:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123776:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763213.869245:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123904:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763213.869267:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763213.869278:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763213/real 1789763213] req@ffff8eb650f81f80 x1876699494123904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763213.869293:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494123904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763213.869303:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763213.869307:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763213.869311:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494123904:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763213.869318:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494124032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763213.869336:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763213.869347:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763213/real 1789763213] req@ffff8eb650f83800 x1876699494124032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763213.869362:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494124032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763213.869379:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763213.869385:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763213.869463:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494124032:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763214.572635:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763215.596648:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763216.620598:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763217.644607:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763218.668659:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763218.988534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763218.988548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763218.988553:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763218.988566:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3223 00000100:00080000:3.0:1789763218.988579:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763218.988588:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763218.988635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494124160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763218.988673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763218.988679:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763218.988681:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763218.988686:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3223 00000100:00080000:3.0:1789763218.988690:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763218.988695:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763218.988706:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763218.988722:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494124288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763218.988737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763218.988742:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763218.988744:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763218.988750:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3223 00000100:00080000:3.0:1789763218.988754:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763218.988773:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763218.988780:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763218.988792:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494124160:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763218.988804:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494124416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763218.988823:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763218.988829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763218.988832:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763218.988858:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3223 00000100:00080000:3.0:1789763218.988863:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763218.988870:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763218.988877:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763218.988891:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494124544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763218.988967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763218.989038:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763218/real 1789763218] req@ffff8eb645148000 x1876699494124160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763218.989068:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494124160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763218.989085:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763218.989091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763218.989097:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494124160:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763218.989109:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763218.989138:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763218.989153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763218/real 1789763218] req@ffff8eb645149880 x1876699494124288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763218.989168:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494124288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763218.989179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763218.989185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763218.989189:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763218.989227:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124416:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763218.989249:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763218.989262:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763218/real 1789763218] req@ffff8eb64514b100 x1876699494124416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763218.989277:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494124416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763218.989287:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763218.989290:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763218.989294:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763218.989303:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494124544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763218.989323:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763218.989334:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763218/real 1789763218] req@ffff8eb64514bb80 x1876699494124544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763218.989351:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494124544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763218.989367:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763218.989374:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763218.989381:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494124544:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763219.692597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763220.716641:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763221.740621:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763222.764625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763223.788545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763224.108520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763224.108534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763224.108540:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763224.108554:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3228 00000100:00080000:3.0:1789763224.108567:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763224.108576:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763224.108644:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494124672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763224.108691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763224.108699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763224.108702:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763224.108709:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3228 00000100:00080000:3.0:1789763224.108716:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763224.108725:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763224.108742:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763224.108760:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494124672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763224.108776:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494124800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763224.108829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763224.108838:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763224.108842:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763224.108847:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3228 00000100:00080000:3.0:1789763224.108856:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763224.108863:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763224.108873:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763224.108897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763224.108899:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494124928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763224.108919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763224.108924:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763224.108927:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763224.108935:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3228 00000100:00080000:3.0:1789763224.108940:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763224.108946:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763224.108954:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763224.108963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763224] req@ffff8eb64514b800 x1876699494124672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763224.108975:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494125056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763224.109012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494124672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763224.109042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763224.109049:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763224.109056:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494124672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763224.109088:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763224.109118:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763224.109134:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763224] req@ffff8eb645148e00 x1876699494124800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763224.109150:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494124800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763224.109160:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763224.109165:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763224.109169:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124800:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763224.109176:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124928:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763224.109196:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763224.109207:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763224] req@ffff8eb645148380 x1876699494124928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763224.109221:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494124928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763224.109231:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763224.109235:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763224.109238:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494124928:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763224.109245:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494125056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763224.109264:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763224.109275:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763224/real 1789763224] req@ffff8eb64514a300 x1876699494125056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763224.109289:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494125056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763224.109299:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763224.109302:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763224.109305:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494125056:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763224.812696:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763225.836629:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763226.860655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763227.884593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763228.908653:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763229.228592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763229.228605:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763229.228610:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763229.228623:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3233 00000100:00080000:0.0:1789763229.228636:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763229.228656:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763229.228705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494125184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763229.228745:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763229.228753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763229.228756:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763229.228794:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3233 00000100:00080000:0.0:1789763229.228800:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763229.228806:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763229.228818:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763229.228835:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494125312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763229.228850:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763229.228854:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763229.228857:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763229.228862:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3233 00000100:00080000:0.0:1789763229.228866:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763229.228874:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763229.228879:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494125184:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789763229.228883:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763229.228904:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494125440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763229.228919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763229.228924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763229.228926:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763229.228931:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3233 00000100:00080000:0.0:1789763229.228936:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763229.228940:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763229.228948:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763229.228962:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494125568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763229.228983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763229.229052:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff8eb64304dc00 x1876699494125184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763229.229078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494125184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763229.229095:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763229.229101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763229.229110:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494125184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763229.229131:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763229.229159:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763229.229175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff8eb64304ce00 x1876699494125312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763229.229190:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494125312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763229.229201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763229.229205:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763229.229209:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125312:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763229.229239:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125440:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763229.229260:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763229.229271:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff8eb64304f800 x1876699494125440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763229.229285:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494125440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763229.229301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763229.229305:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763229.229309:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763229.229316:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494125568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763229.229334:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763229.229345:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763229/real 1789763229] req@ffff8eb64304c380 x1876699494125568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763229.229359:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494125568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763229.229373:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763229.229380:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763229.229386:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494125568:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763229.932590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763230.956594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763231.980524:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763233.004551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763234.028508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763234.348542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763234.348562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763234.348569:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763234.348585:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3238 00000100:00080000:0.0:1789763234.348599:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763234.348609:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763234.348661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494125696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763234.348706:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763234.348714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763234.348718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763234.348751:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3238 00000100:00080000:0.0:1789763234.348761:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763234.348771:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763234.348788:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763234.348790:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494125696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763234.348816:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494125824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763234.348834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763234.348839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763234.348842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763234.348847:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3238 00000100:00080000:0.0:1789763234.348867:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763234.348873:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763234.348880:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763234.348883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763234.348905:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494125952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763234.348920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763234.348925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763234.348927:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763234.348935:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3238 00000100:00080000:0.0:1789763234.348940:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763234.348945:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff8eb64304d500 x1876699494125696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763234.348951:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763234.348964:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763234.348988:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494126080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763234.348996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494125696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763234.349023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763234.349032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763234.349042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494125696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763234.349082:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763234.349133:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763234.349153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff8eb64304d880 x1876699494125824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763234.349171:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494125824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763234.349183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763234.349187:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763234.349192:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763234.349200:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125952:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763234.349225:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763234.349236:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff8eb64304f480 x1876699494125952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763234.349251:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494125952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763234.349261:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763234.349264:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763234.349268:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494125952:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763234.349275:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494126080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763234.349306:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763234.349344:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763234/real 1789763234] req@ffff8eb64304e680 x1876699494126080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763234.349388:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494126080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763234.349459:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763234.349468:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763234.349475:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494126080:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763235.052582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763236.076599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763237.100742:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763238.124590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763239.148651:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763239.468569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763239.468582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763239.468587:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763239.468599:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3243 00000100:00080000:3.0:1789763239.468613:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763239.468624:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763239.468680:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494126208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763239.468754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763239.468758:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763239.468761:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763239.468767:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3243 00000100:00080000:3.0:1789763239.468790:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763239.468797:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763239.468808:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763239.468821:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494126208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763239.468829:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494126336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763239.468845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763239.468849:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763239.468851:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763239.468855:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3243 00000100:00080000:3.0:1789763239.468859:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763239.468864:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763239.468870:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763239.468883:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494126464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763239.468894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763239.468898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763239.468900:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763239.468908:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763239.468912:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3243 00000100:00080000:3.0:1789763239.468919:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763239.468924:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763239.468930:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763239.468944:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494126592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763239.468967:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff8eb645149500 x1876699494126208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763239.468991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494126208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763239.469004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763239.469009:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763239.469016:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494126208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763239.469066:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763239.469104:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763239.469123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff8eb645148a80 x1876699494126336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763239.469137:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494126336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763239.469147:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763239.469151:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763239.469154:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126336:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763239.469160:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126464:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763239.469182:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763239.469192:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff8eb64514a680 x1876699494126464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763239.469204:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494126464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763239.469212:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763239.469215:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763239.469219:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126464:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763239.469224:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494126592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763239.469239:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763239.469253:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763239/real 1789763239] req@ffff8eb645149c00 x1876699494126592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763239.469272:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494126592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763239.469285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763239.469290:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763239.469296:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494126592:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763240.172625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763241.196732:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763242.220685:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763243.244604:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763244.268625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763244.588516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763244.588525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763244.588530:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763244.588539:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3249 00000100:00080000:3.0:1789763244.588550:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763244.588557:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763244.588616:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494126720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763244.588643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763244.588647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763244.588669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763244.588674:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3249 00000100:00080000:3.0:1789763244.588679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763244.588684:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763244.588695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763244.588710:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494126720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763244.588712:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494126848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763244.588732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763244.588736:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763244.588738:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763244.588744:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3249 00000100:00080000:3.0:1789763244.588748:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763244.588753:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763244.588759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763244.588772:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494126976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763244.588782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763244.588786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763244.588788:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763244.588795:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3249 00000800:00000100:1.0:1789763244.588796:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763244.588802:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763244.588808:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763244.588814:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763244.588826:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494127104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763244.588850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff8eb645149c00 x1876699494126720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763244.588888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494126720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763244.588898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763244.588902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763244.588906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494126720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763244.588927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763244.588947:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763244.588958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff8eb64514a680 x1876699494126848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763244.588969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494126848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763244.588981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763244.588985:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763244.588990:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126848:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763244.588998:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126976:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763244.589030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763244.589045:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff8eb645148a80 x1876699494126976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763244.589057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494126976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763244.589064:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763244.589067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763244.589070:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494126976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763244.589075:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494127104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763244.589112:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763244.589121:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763244/real 1789763244] req@ffff8eb645149500 x1876699494127104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763244.589132:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494127104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763244.589140:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763244.589142:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763244.589145:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494127104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763245.292631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763246.316619:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763247.340658:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763248.364586:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763249.388641:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763249.708581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763249.708599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763249.708606:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763249.708642:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3254 00000100:00080000:2.0:1789763249.708656:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763249.708665:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763249.708713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494127232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763249.708764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763249.708769:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763249.708772:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763249.708777:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3254 00000100:00080000:2.0:1789763249.708781:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763249.708786:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763249.708801:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763249.708827:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494127360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763249.708855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763249.708869:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494127232:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763249.708869:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763249.708887:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763249.708898:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3254 00000100:00080000:2.0:1789763249.708906:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763249.708913:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763249.708925:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763249.708947:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494127488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763249.708964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763249.708969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763249.708972:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763249.708977:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3254 00000100:00080000:2.0:1789763249.708985:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763249.708992:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763249.709000:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763249.709013:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763249.709023:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494127616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763249.709112:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff8eb650f80a80 x1876699494127232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763249.709154:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494127232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763249.709177:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763249.709186:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763249.709195:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494127232:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763249.709245:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763249.709298:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763249.709317:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff8eb650f80000 x1876699494127360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763249.709334:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494127360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763249.709346:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763249.709350:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763249.709355:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763249.709362:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127488:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763249.709386:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763249.709444:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff8eb650f80380 x1876699494127488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763249.709461:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494127488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763249.709480:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763249.709484:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763249.709488:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127488:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763249.709496:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494127616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763249.709521:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763249.709533:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763249/real 1789763249] req@ffff8eb650f81500 x1876699494127616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763249.709548:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494127616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763249.709558:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763249.709561:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763249.709565:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494127616:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763250.412640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763251.436668:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763252.460630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763253.484652:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763254.508633:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763254.828543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763254.828557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763254.828563:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763254.828598:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3259 00000100:00080000:3.0:1789763254.828611:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763254.828620:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763254.828666:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494127744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763254.828703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763254.828708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763254.828711:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763254.828717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3259 00000100:00080000:3.0:1789763254.828721:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763254.828726:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763254.828737:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763254.828755:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494127872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763254.828769:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763254.828774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763254.828777:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763254.828781:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3259 00000100:00080000:3.0:1789763254.828785:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763254.828794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494127744:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763254.828796:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763254.828808:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763254.828823:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494128000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763254.828840:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763254.828845:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763254.828848:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763254.828853:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3259 00000100:00080000:3.0:1789763254.828858:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763254.828863:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763254.828868:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763254.828884:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494128128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763254.828912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763254.828964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff8eb64514a300 x1876699494127744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763254.828989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494127744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763254.829004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763254.829010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763254.829015:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494127744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763254.829025:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763254.829050:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763254.829064:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff8eb645148380 x1876699494127872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763254.829079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494127872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763254.829090:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763254.829094:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763254.829098:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494127872:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763254.829126:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128000:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763254.829158:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763254.829178:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff8eb645148e00 x1876699494128000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763254.829203:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494128000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763254.829220:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763254.829227:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763254.829235:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128000:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763254.829247:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494128128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763254.829283:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763254.829299:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763254/real 1789763254] req@ffff8eb64514b800 x1876699494128128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763254.829315:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494128128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763254.829325:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763254.829329:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763254.829332:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494128128:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763255.532650:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763256.556685:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763257.580646:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763258.604614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763259.628688:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763259.948614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763259.948648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763259.948655:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763259.948670:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3264 00000100:00080000:2.0:1789763259.948685:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763259.948694:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763259.948744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494128256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763259.948783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763259.948791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763259.948794:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763259.948799:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3264 00000100:00080000:2.0:1789763259.948804:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763259.948810:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763259.948822:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763259.948839:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494128384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763259.948851:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494128256:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763259.948860:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763259.948866:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763259.948869:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763259.948875:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3264 00000100:00080000:2.0:1789763259.948879:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763259.948883:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763259.948890:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763259.948908:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494128512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763259.948920:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763259.948925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763259.948928:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763259.948933:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3264 00000100:00080000:2.0:1789763259.948938:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763259.948946:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763259.948954:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763259.948959:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763259.948977:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494128640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763259.949017:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763259/real 1789763259] req@ffff8eb650f81500 x1876699494128256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763259.949046:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494128256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763259.949061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763259.949067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763259.949072:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494128256:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763259.949118:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763259.949145:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763259.949178:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763259/real 1789763259] req@ffff8eb650f80380 x1876699494128384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763259.949205:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494128384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763259.949216:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763259.949220:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763259.949224:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763259.949232:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128512:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763259.949269:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763259.949279:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763259/real 1789763259] req@ffff8eb650f80000 x1876699494128512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763259.949292:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494128512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763259.949301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763259.949304:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763259.949307:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763259.949316:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494128640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763259.949460:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763259.949480:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763259/real 1789763259] req@ffff8eb650f80a80 x1876699494128640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763259.949496:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494128640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763259.949513:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763259.949517:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763259.949521:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494128640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763260.652606:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763261.676619:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763262.700615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763263.724618:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763264.748633:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763265.068637:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763265.068650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763265.068657:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763265.068672:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3269 00000100:00080000:0.0:1789763265.068686:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763265.068698:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763265.068746:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494128768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763265.068778:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763265.068783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763265.068786:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763265.068791:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3269 00000100:00080000:0.0:1789763265.068795:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763265.068799:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763265.068811:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763265.068826:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494128896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763265.068844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763265.068848:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763265.068851:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763265.068859:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3269 00000100:00080000:0.0:1789763265.068864:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763265.068865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494128768:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763265.068870:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763265.068881:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763265.068894:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494129024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763265.068916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763265.068920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763265.068922:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763265.068927:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3269 00000100:00080000:0.0:1789763265.068931:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763265.068936:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763265.068942:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763265.068955:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494129152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763265.068983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763265.069050:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff8eb64304ea00 x1876699494128768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763265.069079:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494128768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763265.069094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763265.069099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763265.069105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494128768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763265.069147:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763265.069191:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763265.069218:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff8eb64304c700 x1876699494128896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763265.069246:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494128896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763265.069263:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763265.069270:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763265.069277:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494128896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763265.069290:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129024:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763265.069335:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763265.069354:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff8eb64304f100 x1876699494129024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763265.069370:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494129024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763265.069381:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763265.069385:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763265.069389:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129024:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763265.069452:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494129152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763265.069481:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763265.069496:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763265/real 1789763265] req@ffff8eb64304ca80 x1876699494129152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763265.069543:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494129152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763265.069561:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763265.069567:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763265.069574:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494129152:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763265.772666:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763266.796634:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763267.820673:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763268.844666:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763269.868601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763270.188607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.188623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763270.188629:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.188644:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3274 00000100:00080000:2.0:1789763270.188657:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.188667:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763270.188713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494129280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.188752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.188757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.188759:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.188764:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3274 00000100:00080000:2.0:1789763270.188768:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.188773:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763270.188784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763270.188801:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494129408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.188819:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763270.188829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494129280:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763270.188831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.188841:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.188854:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3274 00000100:00080000:2.0:1789763270.188863:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.188873:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763270.188885:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763270.188912:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494129536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763270.188934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763270.188939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763270.188942:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763270.188949:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3274 00000100:00080000:2.0:1789763270.188956:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763270.188962:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763270.188970:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763270.188982:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:2.0:1789763270.188992:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494129664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763270.189033:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763270.189075:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494129664:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763270.892602:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763271.916654:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763272.940654:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763273.260754:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789763273.260810:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763273.260831:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763273.260882:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763273.260891:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763273.260900:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763273.261014:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763273] req@ffff8eb650f81180 x1876699494129664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763273.261057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494129664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763273.261080:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763273.261089:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763273.261098:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494129664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763273.261113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763273] req@ffff8eb650f81f80 x1876699494129408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763273.261139:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494129408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763273.261154:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763273.261161:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763273.261169:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763273.261182:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763273] req@ffff8eb650f83b80 x1876699494129536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763273.261207:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494129536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763273.261224:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763273.261230:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763273.261237:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763273.261249:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763270/real 1789763273] req@ffff8eb650f83800 x1876699494129280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763273.261273:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494129280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763273.261289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763273.261295:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763273.261302:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494129280:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789763273.964651:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763274.988622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763275.308586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763275.308600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763275.308605:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763275.308620:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3279 00000100:00080000:0.0:1789763275.308633:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763275.308642:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763275.308700:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494129792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763275.308739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763275.308744:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763275.308756:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763275.308762:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3279 00000100:00080000:0.0:1789763275.308766:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763275.308771:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763275.308784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763275.308800:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494129920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763275.308814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763275.308821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763275.308824:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763275.308838:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3279 00000100:00100000:1.0:1789763275.308839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494129792:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763275.308845:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763275.308860:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763275.308873:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763275.308903:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494130048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763275.308925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763275.308932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763275.308937:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763275.308944:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3279 00000100:00080000:0.0:1789763275.308952:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763275.308965:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763275.308969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763275.308981:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763275.309025:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494130176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763275.309061:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff8eb64304ca80 x1876699494129792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763275.309111:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494129792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763275.309132:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763275.309140:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763275.309148:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494129792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763275.309165:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763275.309215:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763275.309243:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff8eb64304f100 x1876699494129920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763275.309269:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494129920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763275.309285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763275.309291:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763275.309298:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494129920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763275.309340:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130048:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763275.309386:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763275.309494:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff8eb64304c700 x1876699494130048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763275.309521:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494130048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763275.309540:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763275.309546:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763275.309553:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130048:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763275.309566:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494130176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763275.309611:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763275.309637:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763275/real 1789763275] req@ffff8eb64304ea00 x1876699494130176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763275.309662:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494130176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763275.309679:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763275.309684:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763275.309691:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494130176:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763276.012661:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763277.036610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763278.060663:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763279.084584:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763280.108638:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763280.428497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763280.428508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763280.428512:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763280.428521:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3284 00000100:00080000:2.0:1789763280.428530:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763280.428536:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763280.428570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494130304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763280.428594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763280.428600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763280.428601:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763280.428605:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3284 00000100:00080000:2.0:1789763280.428608:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763280.428610:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763280.428618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763280.428627:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494130432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763280.428635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763280.428638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763280.428639:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763280.428641:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3284 00000100:00080000:2.0:1789763280.428643:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763280.428646:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763280.428649:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763280.428656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494130560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763280.428681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763280.428685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763280.428688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763280.428694:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3284 00000100:00080000:2.0:1789763280.428698:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763280.428704:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763280.428712:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763280.428722:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494130304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763280.428732:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494130688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763280.428833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763280.428888:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff8eb650f82a00 x1876699494130304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763280.428921:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494130304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763280.428935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763280.428940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763280.428945:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494130304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763280.428954:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763280.428978:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763280.429001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff8eb650f83480 x1876699494130432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763280.429015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494130432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763280.429025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763280.429028:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763280.429032:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763280.429038:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130560:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763280.429064:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763280.429075:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff8eb650f80e00 x1876699494130560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763280.429087:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494130560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763280.429096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763280.429099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763280.429103:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130560:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763280.429131:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494130688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763280.429149:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763280.429159:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763280/real 1789763280] req@ffff8eb650f80700 x1876699494130688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763280.429171:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494130688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763280.429179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763280.429182:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763280.429186:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494130688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763281.132534:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763282.156543:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763283.180637:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763284.204550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763285.228554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763285.548452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.548460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763285.548464:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.548472:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3290 00000100:00080000:2.0:1789763285.548481:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.548486:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763285.548518:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494130816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.548542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.548544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.548545:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.548548:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3290 00000100:00080000:2.0:1789763285.548550:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.548553:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763285.548560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.548568:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494130944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.548575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.548578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.548579:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.548582:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3290 00000100:00080000:2.0:1789763285.548585:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.548588:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763285.548592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763285.548600:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494131072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763285.548607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763285.548609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763285.548610:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763285.548613:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3290 00000100:00080000:2.0:1789763285.548615:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763285.548619:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763285.548630:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494130816:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763285.548631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763285.548651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494131200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763285.548718:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763285.548768:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff8eb650f81880 x1876699494130816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763285.548789:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494130816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763285.548800:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763285.548803:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763285.548807:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494130816:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763285.548814:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763285.548837:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763285.548847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff8eb650f82680 x1876699494130944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763285.548855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494130944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763285.548861:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763285.548863:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763285.548866:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494130944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763285.548870:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131072:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763285.548885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763285.548892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff8eb650f82d80 x1876699494131072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763285.548900:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494131072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763285.548908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763285.548912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763285.548916:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763285.548947:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494131200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763285.548966:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763285.548975:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763285/real 1789763285] req@ffff8eb650f81c00 x1876699494131200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763285.548983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494131200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763285.548989:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763285.548991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763285.548994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494131200:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763286.252489:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763287.276605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763288.300540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763289.324562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763290.348521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763290.668496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763290.668505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763290.668509:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763290.668517:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3295 00000100:00080000:0.0:1789763290.668526:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763290.668531:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763290.668564:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494131328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763290.668585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763290.668588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763290.668589:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763290.668592:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3295 00000100:00080000:0.0:1789763290.668595:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763290.668597:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763290.668604:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763290.668614:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494131456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763290.668622:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763290.668624:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763290.668627:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763290.668630:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3295 00000100:00080000:0.0:1789763290.668633:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763290.668636:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494131328:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763290.668639:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763290.668647:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763290.668655:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494131584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763290.668662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763290.668665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763290.668667:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763290.668670:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3295 00000100:00080000:0.0:1789763290.668673:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763290.668676:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763290.668679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763290.668685:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494131712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763290.668702:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763290.668737:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763290] req@ffff8eb64304e680 x1876699494131328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763290.668756:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494131328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763290.668764:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763290.668767:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763290.668770:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494131328:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763290.668774:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763290.668788:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763290.668796:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763290] req@ffff8eb64304f480 x1876699494131456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763290.668804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494131456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763290.668810:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763290.668812:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763290.668814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131456:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763290.668833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131584:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763290.668844:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763290.668850:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763290] req@ffff8eb64304d880 x1876699494131584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763290.668857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494131584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763290.668867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763290.668869:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763290.668871:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131584:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763290.668875:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494131712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763290.668884:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763290.668890:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763290/real 1789763290] req@ffff8eb64304d500 x1876699494131712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763290.668896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494131712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763290.668902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763290.668904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763290.668906:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494131712:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763291.372509:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763292.396505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763293.420506:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763294.444517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763295.468510:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763295.788464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763295.788475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763295.788480:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763295.788490:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3300 00000100:00080000:0.0:1789763295.788500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763295.788507:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763295.788541:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494131840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763295.788569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763295.788573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763295.788575:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763295.788579:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3300 00000100:00080000:0.0:1789763295.788583:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763295.788588:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763295.788597:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763295.788612:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494131968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763295.788617:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494131840:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763295.788634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763295.788638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763295.788640:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763295.788645:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3300 00000100:00080000:0.0:1789763295.788650:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763295.788654:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763295.788660:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763295.788673:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494132096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763295.788678:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763295.788691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763295.788696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763295.788698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763295.788705:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3300 00000100:00080000:0.0:1789763295.788709:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763295.788714:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763295.788718:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff8eb64304c380 x1876699494131840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763295.788722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763295.788741:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494132224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763295.788743:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494131840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763295.788773:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763295.788779:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763295.788785:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494131840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763295.788806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763295.788825:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763295.788834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff8eb64304f800 x1876699494131968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763295.788843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494131968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763295.788849:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763295.788851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763295.788853:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494131968:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763295.788857:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132096:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763295.788868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763295.788874:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff8eb64304ce00 x1876699494132096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763295.788882:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494132096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763295.788888:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763295.788889:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763295.788891:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763295.788895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494132224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763295.788905:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763295.788911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763295/real 1789763295] req@ffff8eb64304dc00 x1876699494132224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763295.788918:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494132224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763295.788923:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763295.788925:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763295.788927:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494132224:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763296.492498:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763297.516534:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763298.540491:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763299.564521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763300.588502:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763300.908486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763300.908496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763300.908499:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763300.908507:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3305 00000100:00080000:2.0:1789763300.908516:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763300.908522:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763300.908560:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494132352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763300.908591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763300.908597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763300.908598:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763300.908601:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3305 00000100:00080000:2.0:1789763300.908604:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763300.908608:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763300.908615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763300.908625:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494132480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763300.908632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763300.908635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763300.908636:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763300.908639:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3305 00000100:00080000:2.0:1789763300.908641:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763300.908643:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763300.908647:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763300.908654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494132608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763300.908662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763300.908664:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494132352:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763300.908667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763300.908671:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763300.908675:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3305 00000100:00080000:2.0:1789763300.908677:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763300.908679:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763300.908682:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763300.908691:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494132736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763300.908728:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763300.908771:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763300/real 1789763300] req@ffff8eb650f81c00 x1876699494132352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763300.908785:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494132352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763300.908793:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763300.908796:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763300.908798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494132352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763300.908803:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763300.908819:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763300.908828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763300/real 1789763300] req@ffff8eb650f82d80 x1876699494132480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763300.908837:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494132480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763300.908842:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763300.908844:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763300.908846:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763300.908864:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132608:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763300.908875:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763300.908881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763300/real 1789763300] req@ffff8eb650f82680 x1876699494132608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763300.908888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494132608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763300.908893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763300.908895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763300.908897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132608:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763300.908900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494132736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763300.908909:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763300.908915:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763300/real 1789763300] req@ffff8eb650f81880 x1876699494132736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763300.908922:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494132736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763300.908931:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763300.908933:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763300.908935:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494132736:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763301.612518:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763302.636496:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763303.660525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763304.684508:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763305.708549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763306.028469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763306.028478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763306.028482:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763306.028490:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3310 00000100:00080000:2.0:1789763306.028500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763306.028508:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763306.028547:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494132864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763306.028570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763306.028573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763306.028574:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763306.028577:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3310 00000100:00080000:2.0:1789763306.028580:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763306.028583:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763306.028590:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763306.028600:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494132992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763306.028607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763306.028610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763306.028611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763306.028613:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3310 00000100:00080000:2.0:1789763306.028616:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763306.028621:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763306.028624:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494132864:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763306.028624:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763306.028638:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494133120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763306.028650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763306.028654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763306.028656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763306.028660:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3310 00000100:00080000:2.0:1789763306.028664:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763306.028669:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763306.028675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763306.028686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494133248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763306.028702:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763306.028744:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff8eb650f80700 x1876699494132864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763306.028765:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494132864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763306.028777:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763306.028782:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763306.028786:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494132864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763306.028794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763306.028816:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763306.028828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff8eb650f80e00 x1876699494132992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763306.028842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494132992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763306.028852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763306.028855:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763306.028859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494132992:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763306.028882:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133120:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763306.028904:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763306.028916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff8eb650f83480 x1876699494133120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763306.028929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494133120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763306.028937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763306.028941:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763306.028944:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133120:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763306.028950:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494133248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763306.028969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763306.028981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763306/real 1789763306] req@ffff8eb650f82a00 x1876699494133248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763306.028993:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494133248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763306.029002:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763306.029005:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763306.029008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494133248:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763306.732500:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763307.756509:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763308.780533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763309.804545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763310.828509:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763311.148500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763311.148511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763311.148517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763311.148530:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3315 00000100:00080000:2.0:1789763311.148543:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763311.148553:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763311.148608:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494133376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763311.148645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763311.148654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763311.148657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763311.148662:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3315 00000100:00080000:2.0:1789763311.148666:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763311.148670:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763311.148682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763311.148698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494133504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763311.148714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763311.148721:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763311.148726:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763311.148727:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494133376:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763311.148735:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3315 00000100:00080000:2.0:1789763311.148748:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763311.148763:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763311.148772:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763311.148798:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494133632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763311.148834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789763311.148846:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763311.148847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763311.148857:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763311.148869:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3315 00000100:00080000:2.0:1789763311.148880:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763311.148888:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763311.148912:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763311.148921:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff8eb650f83800 x1876699494133376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763311.148947:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494133760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763311.148955:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494133376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763311.149009:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763311.149018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763311.149025:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494133376:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763311.149066:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763311.149093:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763311.149108:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff8eb650f83b80 x1876699494133504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763311.149124:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494133504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763311.149135:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763311.149139:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763311.149143:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763311.149151:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133632:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763311.149211:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763311.149231:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff8eb650f81f80 x1876699494133632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763311.149268:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494133632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763311.149285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763311.149292:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763311.149298:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494133632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763311.149310:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494133760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763311.149349:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763311.149372:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763311/real 1789763311] req@ffff8eb650f81180 x1876699494133760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763311.149455:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494133760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763311.149467:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763311.149471:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763311.149475:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494133760:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763311.852635:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763312.876617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763313.900569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763314.924621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763315.948619:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763316.268481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.268490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763316.268494:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.268502:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3320 00000100:00080000:2.0:1789763316.268511:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.268517:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763316.268549:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494133888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763316.268572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.268575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.268576:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.268579:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3320 00000100:00080000:2.0:1789763316.268582:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.268584:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763316.268592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.268601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494134016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763316.268608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.268611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.268613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763316.268618:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494133888:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763316.268618:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3320 00000100:00080000:2.0:1789763316.268624:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.268629:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763316.268633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763316.268642:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494134144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763316.268649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763316.268653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763316.268654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763316.268657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3320 00000100:00080000:2.0:1789763316.268662:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763316.268665:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763316.268668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763316.268676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494134272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763316.268687:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763316.268748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff8eb650f80a80 x1876699494133888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763316.268766:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494133888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763316.268775:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763316.268778:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763316.268792:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494133888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763316.268810:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763316.268825:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763316.268833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff8eb650f80000 x1876699494134016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763316.268841:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494134016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763316.268857:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763316.268859:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763316.268861:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763316.268865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134144:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763316.268875:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763316.268881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff8eb650f80380 x1876699494134144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763316.268888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494134144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763316.268894:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763316.268895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763316.268897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134144:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763316.268901:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494134272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763316.268911:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763316.268926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763316/real 1789763316] req@ffff8eb650f81500 x1876699494134272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763316.268933:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494134272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763316.268937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763316.268939:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763316.268941:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494134272:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763316.972686:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763317.996680:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763319.020617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763320.044630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763321.068521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763321.388462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.388470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763321.388474:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.388482:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3325 00000100:00080000:2.0:1789763321.388490:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.388496:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763321.388527:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494134400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.388548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.388553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.388554:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.388557:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3325 00000100:00080000:2.0:1789763321.388559:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.388562:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763321.388571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.388579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494134528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.388589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763321.388591:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494134400:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763321.388592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.388596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.388602:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3325 00000100:00080000:2.0:1789763321.388605:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.388607:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763321.388610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763321.388618:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494134656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763321.388625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763321.388628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763321.388629:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763321.388633:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3325 00000100:00080000:2.0:1789763321.388637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763321.388639:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763321.388643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763321.388650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494134784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763321.388660:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763321.388700:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff8eb650f81500 x1876699494134400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763321.388716:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494134400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763321.388728:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763321.388733:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763321.388745:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494134400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763321.388771:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763321.388791:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763321.388801:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff8eb650f80380 x1876699494134528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763321.388810:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494134528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763321.388816:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763321.388818:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763321.388821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134528:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763321.388824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134656:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763321.388836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763321.388849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff8eb650f80000 x1876699494134656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763321.388858:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494134656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763321.388864:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763321.388866:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763321.388868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494134656:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763321.388872:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494134784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763321.388883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763321.388889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763321/real 1789763321] req@ffff8eb650f80a80 x1876699494134784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763321.388896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494134784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763321.388902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763321.388904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763321.388907:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494134784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763322.092522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763323.116624:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763324.140565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763325.164612:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763326.188569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763326.508599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.508611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763326.508617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.508629:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3331 00000100:00080000:0.0:1789763326.508643:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.508652:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763326.508705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494134912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.508754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.508762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.508766:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.508775:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3331 00000100:00080000:0.0:1789763326.508783:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.508793:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763326.508821:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763326.508846:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494135040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.508863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.508868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.508871:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.508881:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3331 00000100:00100000:1.0:1789763326.508884:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494134912:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763326.508887:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.508899:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763326.508908:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763326.508925:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494135168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763326.508941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763326.508946:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763326.508948:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763326.508955:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3331 00000100:00080000:0.0:1789763326.508973:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763326.508978:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763326.508985:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763326.508989:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763326.509005:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494135296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763326.509079:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff8eb64304ed80 x1876699494134912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763326.509107:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494134912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763326.509122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763326.509127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763326.509145:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494134912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763326.509155:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763326.509184:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763326.509200:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff8eb64304c000 x1876699494135040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763326.509216:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494135040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763326.509226:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763326.509231:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763326.509235:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763326.509266:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135168:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763326.509287:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763326.509300:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff8eb64304d180 x1876699494135168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763326.509314:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494135168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763326.509324:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763326.509328:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763326.509332:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763326.509351:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494135296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763326.509374:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763326.509386:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763326/real 1789763326] req@ffff8eb64304e300 x1876699494135296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763326.509442:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494135296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763326.509453:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763326.509457:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763326.509461:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494135296:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763327.212641:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763328.236570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763329.260520:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763330.284586:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763331.308664:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763331.628561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763331.628580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763331.628589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763331.628605:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3336 00000100:00080000:0.0:1789763331.628619:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763331.628630:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763331.628701:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494135424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763331.628749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763331.628754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763331.628757:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763331.628763:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3336 00000100:00080000:0.0:1789763331.628770:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763331.628777:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763331.628791:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763331.628812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494135424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763331.628820:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494135552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763331.628847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763331.628855:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763331.628860:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763331.628869:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3336 00000100:00080000:0.0:1789763331.628894:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763331.628904:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763331.628911:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763331.628923:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763331.628957:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494135680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763331.628973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763331.628978:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763331.628980:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763331.628986:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3336 00000100:00080000:0.0:1789763331.628992:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763331.629002:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763331.629002:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff8eb64304e300 x1876699494135424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763331.629041:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763331.629072:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494135808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763331.629087:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494135424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.629115:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763331.629124:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763331.629133:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494135424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763331.629172:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763331.629209:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763331.629226:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff8eb64304d180 x1876699494135552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763331.629243:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494135552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.629254:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763331.629258:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763331.629263:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135552:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763331.629270:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135680:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763331.629291:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763331.629302:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff8eb64304c000 x1876699494135680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763331.629318:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494135680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.629328:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763331.629331:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763331.629335:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494135680:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763331.629342:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494135808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763331.629361:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763331.629372:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763331/real 1789763331] req@ffff8eb64304ed80 x1876699494135808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763331.629387:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494135808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763331.629437:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763331.629440:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763331.629444:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494135808:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763332.332635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763333.356578:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763334.380575:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763335.404599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763336.428689:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763336.748555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763336.748568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763336.748574:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763336.748586:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3341 00000100:00080000:2.0:1789763336.748600:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763336.748627:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763336.748700:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494135936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763336.748741:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763336.748748:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763336.748752:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763336.748760:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3341 00000100:00080000:2.0:1789763336.748769:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763336.748777:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763336.748794:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763336.748797:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494135936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763336.748818:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494136064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763336.748834:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763336.748839:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763336.748841:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763336.748846:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3341 00000100:00080000:2.0:1789763336.748853:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763336.748858:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763336.748866:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763336.748881:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494136192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763336.748901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763336.748907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763336.748928:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763336.748938:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3341 00000100:00080000:2.0:1789763336.748946:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763336.748955:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763336.748963:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136064:192.168.203.160@tcp:38:kworker.0 00000008:00100000:2.0:1789763336.749010:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763336.749048:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494136320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763336.749081:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763336.749132:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494136320:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763337.452510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763338.476543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763339.500629:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763339.820909:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789763339.821018:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763339.821050:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763339.821127:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763339.821146:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763339.821162:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763339.821240:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763339] req@ffff8eb650f83800 x1876699494136320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763339.821298:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494136320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763339.821324:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763339.821334:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763339.821343:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494136320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763339.821360:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763339] req@ffff8eb650f81f80 x1876699494136064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763339.821388:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494136064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763339.821460:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763339.821471:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763339.821481:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136064:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763339.821499:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763339] req@ffff8eb650f83b80 x1876699494136192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763339.821531:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494136192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763339.821550:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763339.821559:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763339.821567:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763339.821583:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763336/real 1789763339] req@ffff8eb650f81180 x1876699494135936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763339.821608:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494135936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763339.821622:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763339.821628:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763339.821634:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494135936:192.168.203.160@tcp:250:kworker.0 00010000:00010000:1.0:1789763340.524667:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763341.548648:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763341.868484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763341.868498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763341.868504:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763341.868518:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3346 00000100:00080000:2.0:1789763341.868545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763341.868556:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763341.868602:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494136448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763341.868648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763341.868658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763341.868662:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763341.868669:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3346 00000100:00080000:2.0:1789763341.868679:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763341.868689:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763341.868709:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763341.868740:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494136448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763341.868744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494136576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763341.868791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763341.868797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763341.868800:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763341.868807:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3346 00000100:00080000:2.0:1789763341.868814:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763341.868820:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:3.0:1789763341.868829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763341.868830:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763341.868857:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494136704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763341.868873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763341.868878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763341.868881:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763341.868890:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3346 00000100:00100000:3.0:1789763341.868891:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763341/real 1789763341] req@ffff8eb650f82a00 x1876699494136448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763341.868897:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763341.868910:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763341.868926:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763341.868974:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494136448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763341.868995:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763341.869000:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494136832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763341.869005:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763341.869019:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494136448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763341.869077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789763341.869132:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763341.869157:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763341/real 1789763341] req@ffff8eb650f83480 x1876699494136576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763341.869182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494136576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763341.869199:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763341.869207:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763341.869214:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789763341.869226:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136704:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789763341.869267:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763341.869290:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763341/real 1789763341] req@ffff8eb650f80e00 x1876699494136704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763341.869314:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494136704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763341.869342:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763341.869350:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763341.869357:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494136704:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789763341.869369:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494136832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789763341.869479:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763341.869504:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763341/real 1789763341] req@ffff8eb650f80700 x1876699494136832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763341.869527:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494136832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763341.869541:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763341.869547:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763341.869553:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494136832:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763342.572608:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763343.596557:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763344.620625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763345.644655:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763346.668562:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763346.988506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763346.988516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763346.988521:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763346.988532:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3351 00000100:00080000:0.0:1789763346.988550:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763346.988558:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763346.988599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494136960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763346.988627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763346.988629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763346.988631:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763346.988634:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3351 00000100:00080000:0.0:1789763346.988636:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763346.988640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763346.988649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763346.988661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494137088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763346.988663:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494136960:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763346.988678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763346.988681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763346.988682:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763346.988685:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3351 00000100:00080000:0.0:1789763346.988686:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763346.988689:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763346.988693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763346.988700:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494137216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763346.988707:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763346.988710:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763346.988711:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763346.988713:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3351 00000100:00080000:0.0:1789763346.988716:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763346.988719:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763346.988722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763346.988729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494137344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763346.988743:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763346.988783:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763346/real 1789763346] req@ffff8eb64304dc00 x1876699494136960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763346.988798:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494136960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763346.988808:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763346.988811:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763346.988814:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494136960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763346.988837:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789763346.988852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763346.988860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763346/real 1789763346] req@ffff8eb64304ce00 x1876699494137088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763346.988873:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494137088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763346.988878:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763346.988880:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763346.988882:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137088:192.168.203.160@tcp:38:kworker.0 00000100:00100000:3.0:1789763346.988886:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137216:192.168.203.160@tcp:8:kworker.0 00000800:00000100:3.0:1789763346.988914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763346.988923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763346/real 1789763346] req@ffff8eb64304f800 x1876699494137216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763346.988931:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494137216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763346.988937:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763346.988940:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763346.988942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137216:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789763346.988945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494137344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:3.0:1789763346.988957:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763346.988964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763346/real 1789763346] req@ffff8eb64304c380 x1876699494137344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763346.988971:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494137344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763346.988981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763346.988983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:3.0:1789763346.988985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494137344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763347.692559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763348.716655:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763349.740709:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763350.764562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763351.788553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763352.108537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763352.108550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763352.108555:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763352.108566:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3356 00000100:00080000:3.0:1789763352.108579:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763352.108594:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763352.108635:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494137472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763352.108667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763352.108670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763352.108693:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763352.108700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3356 00000100:00080000:3.0:1789763352.108706:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763352.108712:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763352.108724:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763352.108745:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494137600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763352.108765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763352.108774:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494137472:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763352.108774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763352.108781:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763352.108794:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3356 00000100:00080000:3.0:1789763352.108800:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763352.108804:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763352.108810:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763352.108829:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494137728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763352.108842:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763352.108847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763352.108849:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763352.108855:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3356 00000100:00080000:3.0:1789763352.108902:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763352.108906:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763352.108910:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763352.108920:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494137856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763352.108928:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763352.108973:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff8eb64514bb80 x1876699494137472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763352.108990:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494137472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.108999:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763352.109002:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763352.109006:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494137472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763352.109026:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763352.109043:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763352.109053:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff8eb64514b100 x1876699494137600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763352.109062:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494137600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.109069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763352.109071:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763352.109073:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763352.109078:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137728:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763352.109090:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763352.109096:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff8eb645149880 x1876699494137728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763352.109104:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494137728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.109110:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763352.109112:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763352.109114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494137728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763352.109117:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494137856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763352.109128:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763352.109158:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763352/real 1789763352] req@ffff8eb645148000 x1876699494137856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763352.109172:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494137856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763352.109183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763352.109187:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763352.109191:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494137856:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763352.812609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763353.836530:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763354.860617:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763355.884552:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763356.908646:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763357.228571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763357.228587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763357.228595:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763357.228612:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3361 00000100:00080000:0.0:1789763357.228629:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763357.228641:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763357.228702:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494137984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763357.228744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763357.228749:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763357.228751:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763357.228757:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3361 00000100:00080000:0.0:1789763357.228761:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763357.228766:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763357.228780:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763357.228798:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494138112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763357.228813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763357.228842:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763357.228846:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763357.228853:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3361 00000100:00080000:0.0:1789763357.228858:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763357.228864:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763357.228873:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763357.228900:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494138240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763357.228900:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494137984:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763357.228939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763357.228947:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763357.228951:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763357.228960:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3361 00000100:00080000:0.0:1789763357.228970:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763357.228980:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763357.228992:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763357.229016:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494138368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763357.229050:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763357.229148:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763357] req@ffff8eb64304d500 x1876699494137984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763357.229195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494137984/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763357.229219:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763357.229228:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763357.229237:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494137984:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763357.229253:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763357.229302:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763357.229329:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763357] req@ffff8eb64304d880 x1876699494138112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763357.229354:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494138112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763357.229370:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763357.229377:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763357.229384:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138112:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763357.229505:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138240:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763357.229552:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763357.229577:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763357] req@ffff8eb64304f480 x1876699494138240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763357.229622:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494138240/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763357.229639:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763357.229646:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763357.229653:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138240:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763357.229666:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494138368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763357.229708:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763357.229731:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763357/real 1789763357] req@ffff8eb64304e680 x1876699494138368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763357.229754:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494138368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763357.229770:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763357.229778:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763357.229785:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494138368:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763357.932613:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763358.956690:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763359.980731:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763361.004785:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763362.028726:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763362.348629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763362.348660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763362.348666:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763362.348679:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3366 00000100:00080000:3.0:1789763362.348693:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763362.348701:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763362.348769:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494138496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763362.348862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763362.348869:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763362.348873:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763362.348882:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3366 00000100:00080000:3.0:1789763362.348890:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763362.348899:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763362.348933:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763362.348951:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494138496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763362.348965:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494138624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763362.348984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763362.348989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763362.348993:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763362.348998:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3366 00000100:00080000:3.0:1789763362.349004:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763362.349009:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763362.349019:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763362.349038:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494138752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763362.349045:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763362.349064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763362.349069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763362.349072:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763362.349081:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3366 00000100:00080000:3.0:1789763362.349085:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763362.349091:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763362.349097:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763362.349147:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff8eb64514b480 x1876699494138496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763362.349148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494138880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763362.349222:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494138496/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.349236:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.349242:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763362.349263:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494138496:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763362.349294:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763362.349343:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763362.349360:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff8eb645149180 x1876699494138624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.349377:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494138624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.349388:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.349392:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763362.349446:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138624:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763362.349455:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138752:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763362.349502:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763362.349516:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff8eb64514ad80 x1876699494138752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.349532:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494138752/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.349559:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.349563:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763362.349567:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494138752:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763362.349573:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494138880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763362.349596:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763362.349608:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763362/real 1789763362] req@ffff8eb645148700 x1876699494138880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763362.349622:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494138880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763362.349631:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763362.349634:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763362.349638:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494138880:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763363.052642:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763364.076620:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763365.100556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763366.124649:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763367.148605:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763367.468560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.468575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763367.468582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.468598:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3371 00000100:00080000:0.0:1789763367.468614:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.468627:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763367.468686:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494139008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.468717:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.468721:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.468723:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.468727:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3371 00000100:00080000:0.0:1789763367.468731:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.468735:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763367.468763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763367.468780:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494139136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.468811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.468816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.468819:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763367.468824:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3371 00000100:00080000:0.0:1789763367.468829:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.468835:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763367.468842:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763367.468856:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494139264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763367.468878:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763367.468886:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763367.468890:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763367.468890:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494139008:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763367.468901:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3371 00000100:00080000:0.0:1789763367.468911:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763367.468916:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763367.468932:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763367.468944:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494139392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763367.469052:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763367.469142:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff8eb64304ea00 x1876699494139008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.469170:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494139008/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.469187:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.469193:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763367.469198:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494139008:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763367.469216:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.469245:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763367.469278:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff8eb64304c700 x1876699494139136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.469296:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494139136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.469307:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.469312:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763367.469317:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139136:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763367.469353:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139264:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763367.469375:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763367.469386:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff8eb64304f100 x1876699494139264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.469396:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494139264/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.469404:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.469407:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763367.469439:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139264:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763367.469445:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494139392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763367.469462:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763367.469472:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763367/real 1789763367] req@ffff8eb64304ca80 x1876699494139392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763367.469483:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494139392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763367.469490:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763367.469494:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763367.469496:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494139392:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763368.172694:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763369.196664:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763370.220635:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763371.244655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763372.268597:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763372.588553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763372.588564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763372.588569:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763372.588579:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3377 00000100:00080000:3.0:1789763372.588591:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763372.588624:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763372.588669:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494139520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763372.588700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763372.588703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763372.588730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763372.588736:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3377 00000100:00080000:3.0:1789763372.588742:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763372.588766:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763372.588776:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763372.588786:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494139520:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763372.588797:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494139648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763372.588815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763372.588820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763372.588823:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763372.588828:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3377 00000100:00080000:3.0:1789763372.588835:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763372.588841:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763372.588849:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763372.588868:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494139776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763372.588877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763372.588889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763372.588893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763372.588895:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763372.588902:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3377 00000100:00080000:3.0:1789763372.588907:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763372.588912:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763372.588918:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763372.588928:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff8eb645148700 x1876699494139520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763372.588934:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494139904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763372.588974:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494139520/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763372.588986:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763372.588991:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763372.588995:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494139520:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763372.589020:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763372.589048:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763372.589067:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff8eb64514ad80 x1876699494139648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763372.589085:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494139648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763372.589098:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763372.589103:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763372.589110:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139648:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763372.589117:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139776:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763372.589140:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763372.589150:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff8eb645149180 x1876699494139776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763372.589161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494139776/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763372.589169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763372.589171:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763372.589175:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494139776:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763372.589179:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494139904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763372.589193:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763372.589201:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763372/real 1789763372] req@ffff8eb64514b480 x1876699494139904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763372.589211:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494139904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763372.589218:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763372.589220:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763372.589223:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494139904:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763373.292656:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763374.316634:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763375.340616:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763376.364516:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763377.388720:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763377.708572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.708586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763377.708592:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.708606:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3382 00000100:00080000:0.0:1789763377.708633:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.708640:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763377.708692:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494140032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.708717:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.708720:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.708722:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.708725:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3382 00000100:00080000:0.0:1789763377.708728:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.708731:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763377.708739:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763377.708750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494140160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.708760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.708764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.708767:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763377.708767:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494140032:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763377.708774:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3382 00000100:00080000:0.0:1789763377.708778:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.708780:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763377.708784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763377.708797:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494140288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763377.708811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763377.708815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763377.708816:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763377.708819:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3382 00000100:00080000:0.0:1789763377.708822:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763377.708825:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763377.708828:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763377.708835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763377.708839:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494140416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763377.708881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff8eb64304ca80 x1876699494140032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.708899:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494140032/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.708908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.708911:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763377.708931:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494140032:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763377.708966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.709008:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763377.709033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff8eb64304f100 x1876699494140160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.709060:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494140160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.709079:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.709084:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763377.709088:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140160:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763377.709097:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140288:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763377.709132:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763377.709154:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff8eb64304c700 x1876699494140288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.709178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494140288/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.709196:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.709202:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763377.709210:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140288:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763377.709222:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494140416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763377.709252:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763377.709267:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763377/real 1789763377] req@ffff8eb64304ea00 x1876699494140416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763377.709283:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494140416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763377.709294:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763377.709297:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763377.709301:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494140416:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763378.412605:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763379.436731:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763380.460531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763381.484588:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763382.508656:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763382.828496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763382.828509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763382.828514:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763382.828526:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3387 00000100:00080000:0.0:1789763382.828541:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763382.828550:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763382.828596:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494140544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763382.828651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763382.828660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763382.828663:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763382.828668:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3387 00000100:00080000:0.0:1789763382.828673:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763382.828678:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763382.828692:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763382.828713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494140672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763382.828734:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494140544:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763382.828741:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763382.828751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763382.828755:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763382.828764:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3387 00000100:00080000:0.0:1789763382.828774:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763382.828782:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763382.828795:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763382.828820:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494140800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763382.828843:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763382.828844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763382.828857:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763382.828861:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763382.828873:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3387 00000100:00080000:0.0:1789763382.828883:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763382.828891:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763382.828902:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763382.828918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff8eb64304e680 x1876699494140544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763382.828931:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494140928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763382.828969:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494140544/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.828987:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.828994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763382.829001:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494140544:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763382.829038:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.829090:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763382.829108:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff8eb64304f480 x1876699494140672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.829125:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494140672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.829144:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.829149:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763382.829153:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140672:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763382.829161:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140800:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763382.829182:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763382.829194:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff8eb64304d880 x1876699494140800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.829219:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494140800/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.829231:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.829236:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763382.829242:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494140800:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763382.829254:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494140928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763382.829292:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763382.829316:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763382/real 1789763382] req@ffff8eb64304d500 x1876699494140928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763382.829342:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494140928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763382.829356:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763382.829360:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763382.829366:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494140928:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763383.532551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763384.556617:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763385.580677:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763386.604522:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763387.628590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763387.948538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763387.948551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763387.948557:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763387.948571:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3392 00000100:00080000:0.0:1789763387.948586:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763387.948624:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763387.948679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494141056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763387.948729:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763387.948734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763387.948737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763387.948743:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3392 00000100:00080000:0.0:1789763387.948748:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763387.948753:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763387.948764:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763387.948783:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494141184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763387.948798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763387.948803:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763387.948805:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763387.948815:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494141056:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763387.948816:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3392 00000100:00080000:0.0:1789763387.948825:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763387.948834:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763387.948841:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763387.948857:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494141312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763387.948881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763387.948886:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763387.948888:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763387.948893:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3392 00000100:00080000:0.0:1789763387.948898:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763387.948903:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763387.948909:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763387.948914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763387.948930:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494141440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763387.948987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763387/real 1789763387] req@ffff8eb64304c380 x1876699494141056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763387.949007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494141056/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763387.949018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763387.949022:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763387.949026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494141056:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763387.949048:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763387.949067:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763387.949078:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763387/real 1789763387] req@ffff8eb64304f800 x1876699494141184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763387.949089:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494141184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763387.949096:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763387.949099:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763387.949102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141184:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763387.949107:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141312:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763387.949122:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763387.949151:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763387/real 1789763387] req@ffff8eb64304ce00 x1876699494141312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763387.949165:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494141312/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763387.949175:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763387.949178:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763387.949182:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141312:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763387.949189:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494141440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763387.949208:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763387.949219:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763387/real 1789763387] req@ffff8eb64304dc00 x1876699494141440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763387.949233:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494141440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763387.949242:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763387.949246:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763387.949249:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494141440:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763388.652539:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763389.676655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763390.700612:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763391.724638:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763392.748640:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763393.068628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.068636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763393.068640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.068649:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3397 00000100:00080000:0.0:1789763393.068658:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.068664:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763393.068695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494141568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.068714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.068717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.068718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.068721:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3397 00000100:00080000:0.0:1789763393.068724:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.068727:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763393.068736:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763393.068746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494141568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763393.068747:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494141696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.068760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.068763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.068764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.068766:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3397 00000100:00080000:0.0:1789763393.068768:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.068770:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763393.068774:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763393.068781:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494141824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763393.068788:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763393.068790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763393.068791:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763393.068794:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3397 00000100:00080000:0.0:1789763393.068797:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763393.068800:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763393.068803:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763393.068812:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494141952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763393.068813:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763393.068861:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff8eb64304ed80 x1876699494141568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.068879:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494141568/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.068887:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.068890:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763393.068893:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494141568:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763393.068912:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.068927:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763393.068959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff8eb64304c000 x1876699494141696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.068975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494141696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.068985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.068990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763393.068994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141696:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763393.069001:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141824:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763393.069022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763393.069032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff8eb64304d180 x1876699494141824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.069047:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494141824/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.069065:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.069069:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763393.069072:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494141824:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763393.069079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494141952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763393.069096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763393.069107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763393/real 1789763393] req@ffff8eb64304e300 x1876699494141952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763393.069120:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494141952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763393.069130:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763393.069133:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763393.069137:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494141952:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763393.772602:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763394.796606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763395.820642:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763396.844625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763397.868630:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763398.188578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763398.188592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763398.188599:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763398.188613:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3402 00000100:00080000:0.0:1789763398.188627:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763398.188638:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763398.188687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494142080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763398.188725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763398.188730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763398.188732:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763398.188738:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3402 00000100:00080000:0.0:1789763398.188742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763398.188746:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763398.188758:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763398.188775:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494142208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763398.188788:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763398.188793:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763398.188795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763398.188800:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3402 00000100:00080000:0.0:1789763398.188804:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763398.188811:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494142080:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763398.188813:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763398.188828:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763398.188845:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494142336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763398.188869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763398.188874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763398.188877:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763398.188883:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3402 00000100:00080000:0.0:1789763398.188892:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763398.188897:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763398.188903:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763398.188913:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763398.188922:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494142464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763398.188981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff8eb64304e300 x1876699494142080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.189010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494142080/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.189024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.189030:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763398.189037:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494142080:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763398.189046:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.189072:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763398.189087:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff8eb64304d180 x1876699494142208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.189102:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494142208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.189112:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.189116:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763398.189120:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142208:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763398.189149:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142336:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763398.189169:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763398.189180:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff8eb64304c000 x1876699494142336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.189203:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494142336/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.189219:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.189225:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763398.189232:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142336:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763398.189244:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494142464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763398.189285:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763398.189322:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763398/real 1789763398] req@ffff8eb64304ed80 x1876699494142464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763398.189341:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494142464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763398.189352:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763398.189355:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763398.189360:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494142464:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763398.892590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763399.916647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763400.940714:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763401.964549:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763402.988535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763403.308567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763403.308580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763403.308585:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763403.308598:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3407 00000100:00080000:3.0:1789763403.308620:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763403.308629:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763403.308671:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494142592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763403.308703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763403.308708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763403.308711:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763403.308717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3407 00000100:00080000:3.0:1789763403.308722:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763403.308727:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763403.308739:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763403.308756:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494142720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763403.308760:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494142592:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763403.308790:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763403.308795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763403.308798:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763403.308804:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3407 00000100:00080000:3.0:1789763403.308809:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763403.308814:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763403.308821:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763403.308837:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494142848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763403.308858:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763403.308863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763403.308865:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763403.308871:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3407 00000100:00080000:3.0:1789763403.308878:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763403.308881:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142720:192.168.203.160@tcp:38:kworker.0 00000100:00080000:3.0:1789763403.308887:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763403.308896:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763403.308916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142848:192.168.203.160@tcp:8:kworker.0 00000100:00100000:3.0:1789763403.308916:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494142976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763403.308949:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494142976:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763404.012553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763405.036625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763406.060707:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763406.380750:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763406.380804:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763406.380826:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763406.380878:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763406.380899:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763406.380919:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763406.380953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763406] req@ffff8eb645149880 x1876699494142720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763406.380996:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494142720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763406.381021:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763406.381029:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763406.381039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142720:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763406.381058:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763406] req@ffff8eb64514b100 x1876699494142848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763406.381084:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494142848/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763406.381099:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763406.381105:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763406.381111:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494142848:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763406.381123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763406] req@ffff8eb645148000 x1876699494142592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763406.381144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494142592/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763406.381158:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763406.381166:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763406.381172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494142592:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763406.381212:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763403/real 1789763406] req@ffff8eb64514bb80 x1876699494142976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763406.381239:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494142976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763406.381254:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763406.381261:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763406.381268:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494142976:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763407.084574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763408.108524:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763408.428470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763408.428479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763408.428482:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763408.428490:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3412 00000100:00080000:3.0:1789763408.428500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763408.428506:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763408.428534:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494143104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763408.428553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763408.428556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763408.428557:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763408.428561:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3412 00000100:00080000:3.0:1789763408.428563:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763408.428566:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763408.428574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763408.428588:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494143232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763408.428601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763408.428605:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494143104:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763408.428606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763408.428611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763408.428617:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3412 00000100:00080000:3.0:1789763408.428621:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763408.428626:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763408.428631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763408.428643:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494143360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763408.428650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763408.428653:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763408.428654:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763408.428657:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3412 00000100:00080000:3.0:1789763408.428663:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763408.428666:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763408.428669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763408.428673:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763408.428679:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494143488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763408.428711:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff8eb64514b800 x1876699494143104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.428728:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494143104/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.428740:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.428744:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763408.428748:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494143104:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763408.428781:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.428807:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763408.428821:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff8eb645148e00 x1876699494143232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.428834:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494143232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.428842:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.428845:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763408.428847:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143232:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763408.428852:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143360:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763408.428869:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763408.428882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff8eb645148380 x1876699494143360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.428894:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494143360/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.428903:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.428905:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763408.428909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143360:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763408.428914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494143488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763408.428935:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763408.428946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763408/real 1789763408] req@ffff8eb64514a300 x1876699494143488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763408.428961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494143488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763408.428969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763408.428972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763408.428976:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494143488:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763409.132638:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763410.156621:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763411.180622:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763412.204692:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763413.228604:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763413.548543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763413.548557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763413.548562:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763413.548575:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3418 00000100:00080000:3.0:1789763413.548588:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763413.548597:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763413.548658:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494143616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763413.548691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763413.548695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763413.548698:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763413.548702:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3418 00000100:00080000:3.0:1789763413.548706:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763413.548710:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763413.548721:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763413.548736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494143744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763413.548751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763413.548755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763413.548758:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763413.548763:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3418 00000100:00080000:3.0:1789763413.548766:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763413.548771:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763413.548776:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763413.548799:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494143872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763413.548804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494143616:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763413.548815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763413.548835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763413.548837:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763413.548842:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3418 00000100:00080000:3.0:1789763413.548845:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763413.548848:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763413.548852:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763413.548865:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494144000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763413.548982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763413.549077:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff8eb645149500 x1876699494143616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.549119:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494143616/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.549158:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.549167:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763413.549173:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494143616:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763413.549190:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.549245:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763413.549276:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff8eb645148a80 x1876699494143744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.549308:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494143744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.549328:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.549337:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763413.549344:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143744:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763413.549499:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143872:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763413.549567:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763413.549591:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff8eb64514a680 x1876699494143872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.549609:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494143872/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.549632:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.549636:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763413.549642:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494143872:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763413.549656:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494144000:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763413.549685:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763413.549702:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763413/real 1789763413] req@ffff8eb645149c00 x1876699494144000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763413.549719:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494144000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763413.549729:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763413.549733:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763413.549737:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494144000:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763414.252762:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763415.276727:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763416.300636:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763417.324694:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763418.348683:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763418.668536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763418.668550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763418.668556:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763418.668569:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3423 00000100:00080000:3.0:1789763418.668583:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763418.668593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763418.668643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494144128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763418.668691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763418.668696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763418.668699:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763418.668705:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3423 00000100:00080000:3.0:1789763418.668710:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763418.668714:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763418.668725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763418.668741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494144256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763418.668761:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763418.668768:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763418.668773:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763418.668780:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3423 00000100:00080000:3.0:1789763418.668788:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763418.668810:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763418.668821:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763418.668835:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494144128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763418.668851:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494144384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763418.668890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763418.668897:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763418.668901:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763418.668911:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3423 00000100:00080000:3.0:1789763418.668923:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763418.668932:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763418.668939:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763418.668949:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763418.668977:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494144512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763418.669001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff8eb645149c00 x1876699494144128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.669042:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494144128/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.669057:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.669062:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763418.669067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494144128:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763418.669077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144256:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.669107:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763418.669133:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff8eb64514a680 x1876699494144256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.669148:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494144256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.669158:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.669161:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763418.669165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763418.669203:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144384:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763418.669224:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763418.669236:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff8eb645148a80 x1876699494144384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.669250:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494144384/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.669260:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.669263:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763418.669267:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144384:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763418.669274:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494144512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763418.669292:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763418.669303:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763418/real 1789763418] req@ffff8eb645149500 x1876699494144512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763418.669327:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494144512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763418.669336:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763418.669339:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763418.669343:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494144512:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763419.372622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763420.396610:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763421.420590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763422.444639:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763423.468817:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763423.788610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763423.788625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763423.788632:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763423.788649:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3428 00000100:00080000:2.0:1789763423.788678:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763423.788689:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763423.788763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494144640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763423.788811:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763423.788816:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763423.788819:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763423.788824:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3428 00000100:00080000:2.0:1789763423.788829:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763423.788833:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763423.788848:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763423.788865:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494144768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763423.788878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763423.788882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763423.788885:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763423.788888:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3428 00000100:00080000:2.0:1789763423.788893:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763423.788897:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763423.788903:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763423.788922:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494144896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763423.788936:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763423.788940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763423.788943:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763423.788949:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3428 00000100:00080000:2.0:1789763423.788953:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763423.788964:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763423.788968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494144640:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763423.788970:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763423.788990:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494145024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763423.789079:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763423.789162:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763423] req@ffff8eb650f81880 x1876699494144640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763423.789212:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494144640/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763423.789235:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763423.789244:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763423.789252:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494144640:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763423.789268:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763423.789304:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763423.789323:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763423] req@ffff8eb650f82680 x1876699494144768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763423.789375:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494144768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763423.789446:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763423.789451:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763423.789456:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144768:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763423.789464:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144896:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763423.789496:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763423.789521:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763423] req@ffff8eb650f82d80 x1876699494144896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763423.789536:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494144896/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763423.789547:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763423.789551:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763423.789555:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494144896:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763423.789594:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494145024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763423.789614:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763423.789626:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763423/real 1789763423] req@ffff8eb650f81c00 x1876699494145024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763423.789640:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494145024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763423.789650:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763423.789653:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763423.789658:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494145024:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763424.492619:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763425.516701:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763426.540721:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763427.564719:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763428.588635:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763428.908524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763428.908538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763428.908544:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763428.908557:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3433 00000100:00080000:0.0:1789763428.908572:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763428.908581:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763428.908640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494145152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763428.908675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763428.908680:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763428.908683:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763428.908688:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3433 00000100:00080000:0.0:1789763428.908692:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763428.908697:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763428.908714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763428.908744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494145280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763428.908769:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494145152:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763428.908774:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763428.908786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763428.908790:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763428.908799:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3433 00000100:00080000:0.0:1789763428.908806:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763428.908812:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763428.908821:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763428.908843:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494145408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763428.908858:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763428.908863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763428.908870:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763428.908871:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763428.908885:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3433 00000100:00080000:0.0:1789763428.908893:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763428.908899:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763428.908907:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763428.908923:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494145536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763428.908953:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763428/real 1789763428] req@ffff8eb64304dc00 x1876699494145152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763428.908983:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494145152/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763428.908998:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763428.909004:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763428.909009:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494145152:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763428.909039:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763428.909066:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763428.909082:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763428/real 1789763428] req@ffff8eb64304ce00 x1876699494145280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763428.909098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494145280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763428.909108:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763428.909112:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763428.909116:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145280:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763428.909123:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145408:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763428.909146:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763428.909157:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763428/real 1789763428] req@ffff8eb64304f800 x1876699494145408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763428.909171:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494145408/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763428.909181:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763428.909184:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763428.909187:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145408:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763428.909194:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494145536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763428.909212:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763428.909223:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763428/real 1789763428] req@ffff8eb64304c380 x1876699494145536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763428.909236:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494145536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763428.909246:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763428.909249:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763428.909252:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494145536:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763429.612602:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763430.636595:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763431.660645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763432.684588:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763433.708599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763434.028508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763434.028523:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763434.028529:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763434.028542:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3438 00000100:00080000:0.0:1789763434.028555:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763434.028565:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763434.028623:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494145664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763434.028677:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763434.028685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763434.028689:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763434.028698:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3438 00000100:00080000:0.0:1789763434.028706:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763434.028713:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763434.028727:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763434.028748:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494145664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763434.028774:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494145792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763434.028802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763434.028810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763434.028814:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763434.028823:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3438 00000100:00080000:0.0:1789763434.028832:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763434.028839:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763434.028849:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763434.028860:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763434.028884:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494145920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763434.028901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763434.028904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff8eb64304d500 x1876699494145664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763434.028909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763434.028917:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763434.028933:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3438 00000100:00080000:0.0:1789763434.028944:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763434.028950:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763434.028958:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763434.028967:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494145664/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763434.028978:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494146048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763434.029019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763434.029031:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763434.029039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494145664:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763434.029073:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763434.029109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763434.029127:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff8eb64304d880 x1876699494145792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763434.029144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494145792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.029166:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763434.029170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763434.029174:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145792:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763434.029181:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145920:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763434.029203:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763434.029215:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff8eb64304f480 x1876699494145920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763434.029229:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494145920/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.029247:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763434.029251:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763434.029283:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494145920:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763434.029293:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494146048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763434.029330:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763434.029344:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763434/real 1789763434] req@ffff8eb64304e680 x1876699494146048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763434.029359:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494146048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763434.029370:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763434.029374:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763434.029378:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494146048:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763434.732561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763435.756678:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763436.780534:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763437.804637:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763438.828592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763439.148579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763439.148592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763439.148597:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763439.148611:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3443 00000100:00080000:3.0:1789763439.148625:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763439.148634:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763439.148681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494146176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763439.148720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763439.148728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763439.148730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763439.148736:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3443 00000100:00080000:3.0:1789763439.148740:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763439.148745:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763439.148758:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763439.148775:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494146304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763439.148793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494146176:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763439.148796:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763439.148812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763439.148816:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763439.148823:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3443 00000100:00080000:3.0:1789763439.148827:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763439.148831:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763439.148838:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763439.148855:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494146432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763439.148868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763439.148873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763439.148875:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763439.148881:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3443 00000100:00080000:3.0:1789763439.148887:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763439.148891:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763439.148899:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763439.148918:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494146560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763439.148921:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763439.149000:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff8eb64514a300 x1876699494146176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.149028:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494146176/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.149043:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.149050:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763439.149055:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494146176:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763439.149085:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.149115:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763439.149131:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff8eb645148380 x1876699494146304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.149147:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494146304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.149157:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.149161:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763439.149166:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146304:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763439.149172:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146432:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763439.149192:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763439.149203:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff8eb645148e00 x1876699494146432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.149216:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494146432/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.149257:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.149262:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763439.149266:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146432:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763439.149274:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494146560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763439.149302:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763439.149317:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763439/real 1789763439] req@ffff8eb64514b800 x1876699494146560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763439.149332:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494146560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763439.149342:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763439.149346:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763439.149350:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494146560:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763439.852599:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763440.876628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763441.900619:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763442.924635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763443.948642:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763444.268539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763444.268552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763444.268558:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763444.268584:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3448 00000100:00080000:0.0:1789763444.268598:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763444.268608:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763444.268656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494146688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763444.268694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763444.268699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763444.268701:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763444.268707:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3448 00000100:00080000:0.0:1789763444.268712:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763444.268717:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763444.268728:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763444.268743:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494146816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763444.268756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763444.268760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763444.268762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763444.268766:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3448 00000100:00080000:0.0:1789763444.268770:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763444.268775:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763444.268781:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763444.268794:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494146944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763444.268809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763444.268817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789763444.268820:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494146688:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763444.268821:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763444.268839:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3448 00000100:00080000:0.0:1789763444.268848:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763444.268856:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763444.268866:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763444.268890:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494147072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763444.268963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763444.269019:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff8eb64304ea00 x1876699494146688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.269045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494146688/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.269059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.269065:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763444.269070:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494146688:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763444.269080:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.269109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763444.269123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff8eb64304c700 x1876699494146816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.269148:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494146816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.269158:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.269162:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763444.269166:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146816:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763444.269231:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146944:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763444.269261:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763444.269275:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff8eb64304f100 x1876699494146944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.269290:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494146944/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.269301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.269305:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763444.269309:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494146944:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763444.269316:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494147072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763444.269335:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763444.269346:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763444/real 1789763444] req@ffff8eb64304ca80 x1876699494147072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763444.269359:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494147072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763444.269377:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763444.269381:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763444.269385:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494147072:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763444.972592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763445.996622:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763447.020610:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763448.044620:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763449.068686:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763449.388468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763449.388477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763449.388481:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763449.388490:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3453 00000100:00080000:0.0:1789763449.388500:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763449.388507:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763449.388549:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494147200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763449.388571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763449.388574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763449.388575:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763449.388578:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3453 00000100:00080000:0.0:1789763449.388581:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763449.388584:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763449.388592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763449.388602:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494147328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763449.388611:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494147200:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763449.388615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763449.388619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763449.388621:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763449.388624:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3453 00000100:00080000:0.0:1789763449.388627:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763449.388631:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763449.388635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763449.388645:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494147456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763449.388654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763449.388657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763449.388658:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763449.388662:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3453 00000100:00080000:0.0:1789763449.388665:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763449.388668:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763449.388671:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763449.388677:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763449.388682:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494147584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763449.388727:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff8eb64304ca80 x1876699494147200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.388745:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494147200/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.388761:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.388767:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763449.388773:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494147200:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763449.388802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.388823:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763449.388833:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff8eb64304f100 x1876699494147328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.388843:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494147328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.388849:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.388852:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763449.388855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147328:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763449.388859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147456:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763449.388872:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763449.388879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff8eb64304c700 x1876699494147456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.388887:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494147456/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.388893:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.388895:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763449.388897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147456:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763449.388901:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494147584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763449.388912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763449.388918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763449/real 1789763449] req@ffff8eb64304ea00 x1876699494147584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763449.388927:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494147584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763449.388932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763449.388934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763449.388936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494147584:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763450.092578:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763451.116570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763452.140641:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763453.164623:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763454.188608:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763454.508518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763454.508527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763454.508531:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763454.508541:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3459 00000100:00080000:2.0:1789763454.508551:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763454.508557:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763454.508591:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494147712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763454.508633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763454.508640:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763454.508643:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763454.508651:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3459 00000100:00080000:2.0:1789763454.508657:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763454.508664:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763454.508679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763454.508701:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494147840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763454.508704:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494147712:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763454.508719:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763454.508724:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763454.508727:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763454.508732:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3459 00000100:00080000:2.0:1789763454.508738:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763454.508743:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763454.508751:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763454.508766:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494147968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763454.508786:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763454.508808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763454.508811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763454.508813:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763454.508817:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3459 00000100:00080000:2.0:1789763454.508831:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763454.508835:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763454.508841:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763454.508845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff8eb650f81c00 x1876699494147712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763454.508852:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494148096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763454.508881:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494147712/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.508898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.508904:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763454.508911:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494147712:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763454.508945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763454.508976:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763454.508991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff8eb650f82d80 x1876699494147840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.509005:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494147840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.509014:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.509018:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763454.509021:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147840:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763454.509027:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147968:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763454.509045:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763454.509054:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff8eb650f82680 x1876699494147968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.509067:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494147968/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.509091:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.509094:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763454.509097:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494147968:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763454.509103:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494148096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763454.509145:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763454.509173:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763454/real 1789763454] req@ffff8eb650f81880 x1876699494148096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763454.509182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494148096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763454.509187:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763454.509190:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763454.509192:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494148096:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763455.212632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763456.236607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763457.260655:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763458.284587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763459.308549:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763459.628516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763459.628527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763459.628532:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763459.628543:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3464 00000100:00080000:0.0:1789763459.628555:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763459.628563:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763459.628604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494148224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763459.628637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763459.628641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763459.628643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763459.628647:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3464 00000100:00080000:0.0:1789763459.628650:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763459.628654:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763459.628664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763459.628677:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494148352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763459.628687:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763459.628690:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763459.628692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763459.628695:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3464 00000100:00080000:0.0:1789763459.628697:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763459.628700:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763459.628705:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763459.628715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494148480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763459.628726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763459.628730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763459.628732:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763459.628736:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3464 00000100:00080000:0.0:1789763459.628740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763459.628748:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763459.628749:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494148224:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789763459.628753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763459.628766:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494148608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763459.628830:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763459.628904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff8eb64304e680 x1876699494148224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.628935:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494148224/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.628952:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.628958:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763459.628964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494148224:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763459.628973:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.629001:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763459.629016:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff8eb64304f480 x1876699494148352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.629031:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494148352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.629041:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.629045:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763459.629049:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148352:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763459.629056:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148480:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763459.629120:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763459.629131:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff8eb64304d880 x1876699494148480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.629143:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494148480/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.629152:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.629155:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763459.629158:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148480:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763459.629187:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494148608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763459.629205:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763459.629215:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763459/real 1789763459] req@ffff8eb64304d500 x1876699494148608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763459.629226:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494148608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763459.629234:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763459.629237:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763459.629240:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494148608:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763460.332643:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763461.356549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763462.380584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763463.404610:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763464.428585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763464.748566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763464.748580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763464.748587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763464.748604:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3469 00000100:00080000:1.0:1789763464.748623:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763464.748634:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763464.748694:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494148736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763464.748746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763464.748751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763464.748754:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763464.748760:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3469 00000100:00080000:1.0:1789763464.748767:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763464.748773:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763464.748785:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763464.748802:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494148864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763464.748815:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763464.748820:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763464.748822:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763464.748826:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3469 00000100:00080000:1.0:1789763464.748830:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763464.748834:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763464.748841:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763464.748853:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494148992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763464.748885:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494148736:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763464.748886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763464.748896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763464.748902:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763464.748912:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3469 00000100:00080000:1.0:1789763464.748916:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763464.748920:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763464.748933:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763464.748949:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494149120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763464.749011:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763464.749123:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff8eb65016f480 x1876699494148736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763464.749161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494148736/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763464.749180:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763464.749188:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763464.749194:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494148736:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763464.749208:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763464.749250:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763464.749276:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff8eb65016ed80 x1876699494148864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763464.749301:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494148864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763464.749331:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763464.749338:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763464.749344:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148864:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763464.749378:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148992:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763464.749444:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763464.749460:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff8eb65016d180 x1876699494148992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763464.749475:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494148992/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763464.749486:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763464.749490:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763464.749494:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494148992:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763464.749501:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494149120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763464.749521:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763464.749532:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763464/real 1789763464] req@ffff8eb65016c000 x1876699494149120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763464.749546:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494149120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763464.749556:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763464.749559:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763464.749564:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494149120:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763465.452597:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763466.476609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763467.500557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763468.524512:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763469.548639:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763469.868576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763469.868591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763469.868596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763469.868610:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3474 00000100:00080000:0.0:1789763469.868624:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763469.868633:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763469.868682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494149248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763469.868727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763469.868732:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763469.868735:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763469.868740:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3474 00000100:00080000:0.0:1789763469.868744:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763469.868750:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763469.868761:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763469.868779:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494149376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763469.868795:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763469.868801:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763469.868807:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763469.868823:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3474 00000100:00100000:1.0:1789763469.868824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494149248:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763469.868838:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763469.868852:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763469.868859:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763469.868876:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494149504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763469.868889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763469.868894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763469.868897:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763469.868902:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3474 00000100:00080000:0.0:1789763469.868906:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763469.868911:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763469.868920:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763469.868931:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494149632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763469.869070:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149376:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763469.869125:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149504:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763469.869154:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494149632:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763470.572634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763471.596600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763472.620693:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763472.940674:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763472.940731:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763472.940749:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763472.940801:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763472.940814:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763472.940837:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763472.940868:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763469/real 1789763472] req@ffff8eb64304f800 x1876699494149376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763472.940908:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494149376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763472.940925:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763472.940932:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763472.940938:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149376:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763472.940951:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763469/real 1789763472] req@ffff8eb64304ce00 x1876699494149504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763472.940991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494149504/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763472.941004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763472.941010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763472.941017:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149504:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763472.941029:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763469/real 1789763472] req@ffff8eb64304dc00 x1876699494149632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763472.941045:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494149632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763472.941055:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763472.941060:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763472.941065:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494149632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763472.941075:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763469/real 1789763472] req@ffff8eb64304c380 x1876699494149248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763472.941090:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494149248/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763472.941101:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763472.941106:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763472.941111:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494149248:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789763473.644655:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763474.668582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763474.988502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763474.988518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763474.988524:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763474.988540:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3479 00000100:00080000:0.0:1789763474.988556:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763474.988564:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763474.988608:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494149760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763474.988654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763474.988662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763474.988665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763474.988671:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3479 00000100:00080000:0.0:1789763474.988676:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763474.988680:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763474.988693:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763474.988718:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494149888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763474.988738:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763474.988743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763474.988750:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763474.988750:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494149760:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763474.988766:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3479 00000100:00080000:0.0:1789763474.988774:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763474.988781:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763474.988790:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763474.988814:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494150016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763474.988833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763474.988838:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763474.988840:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763474.988844:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3479 00000800:00000100:1.0:1789763474.988853:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763474.988854:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763474.988878:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763474.988889:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763474.988913:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494150144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763474.988941:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763474/real 1789763474] req@ffff8eb64304ed80 x1876699494149760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763474.989007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494149760/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763474.989036:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763474.989043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763474.989050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494149760:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763474.989095:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763474.989136:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763474.989158:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763474/real 1789763474] req@ffff8eb64304c000 x1876699494149888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763474.989178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494149888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763474.989191:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763474.989197:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763474.989203:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494149888:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763474.989213:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150016:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763474.989248:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763474.989268:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763474/real 1789763474] req@ffff8eb64304d180 x1876699494150016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763474.989288:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494150016/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763474.989301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763474.989307:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763474.989312:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150016:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763474.989323:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494150144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763474.989357:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763474.989375:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763474/real 1789763474] req@ffff8eb64304e300 x1876699494150144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763474.989435:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494150144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763474.989450:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763474.989456:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763474.989462:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494150144:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763475.692636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763476.716625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763477.740570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763478.764564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763479.788634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763480.108582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763480.108610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763480.108615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763480.108627:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3484 00000100:00080000:1.0:1789763480.108642:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763480.108651:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763480.108719:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494150272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763480.108774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763480.108778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763480.108780:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763480.108787:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3484 00000100:00080000:1.0:1789763480.108795:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763480.108803:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763480.108820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763480.108829:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494150272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763480.108877:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494150400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763480.108899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763480.108905:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763480.108908:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763480.108941:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3484 00000100:00080000:1.0:1789763480.108974:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763480.108979:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763480.108985:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763480.108991:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763480.109003:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494150528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763480.109016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763480.109021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763480.109023:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763480.109028:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3484 00000100:00080000:1.0:1789763480.109033:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763480.109037:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff8eb65016f100 x1876699494150272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763480.109042:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763480.109054:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763480.109062:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494150272/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763480.109071:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494150656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763480.109075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763480.109084:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763480.109090:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494150272:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763480.109126:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763480.109146:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763480.109155:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff8eb65016ca80 x1876699494150400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763480.109182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494150400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763480.109191:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763480.109194:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763480.109199:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150400:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763480.109205:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150528:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763480.109233:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763480.109249:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff8eb65016c000 x1876699494150528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763480.109268:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494150528/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763480.109282:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763480.109287:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763480.109293:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150528:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763480.109302:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494150656:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763480.109327:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763480.109357:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763480/real 1789763480] req@ffff8eb65016d180 x1876699494150656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763480.109370:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494150656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763480.109378:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763480.109382:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763480.109385:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494150656:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763480.812581:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763481.836551:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763482.860585:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763483.884636:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763484.908650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763485.228521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763485.228533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763485.228538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763485.228549:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3489 00000100:00080000:1.0:1789763485.228562:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763485.228570:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763485.228614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494150784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763485.228649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763485.228654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763485.228656:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763485.228660:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3489 00000100:00080000:1.0:1789763485.228664:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763485.228668:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763485.228678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763485.228708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494150912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763485.228723:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763485.228727:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763485.228731:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763485.228735:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3489 00000100:00080000:1.0:1789763485.228741:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763485.228745:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763485.228753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763485.228766:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494151040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763485.228783:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763485.228787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494150784:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763485.228790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763485.228794:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763485.228815:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3489 00000100:00080000:1.0:1789763485.228821:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763485.228825:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763485.228830:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763485.228850:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494151168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763485.228982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763485.229085:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff8eb65016ed80 x1876699494150784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763485.229112:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494150784/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763485.229129:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763485.229136:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763485.229143:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494150784:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763485.229153:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150912:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763485.229184:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763485.229198:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff8eb65016f480 x1876699494150912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763485.229213:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494150912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763485.229223:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763485.229226:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763485.229230:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494150912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763485.229269:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151040:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763485.229290:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763485.229301:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff8eb65016dc00 x1876699494151040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763485.229323:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494151040/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763485.229333:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763485.229336:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763485.229340:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151040:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763485.229347:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494151168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763485.229365:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763485.229376:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763485/real 1789763485] req@ffff8eb65016df80 x1876699494151168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763485.229389:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494151168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763485.229471:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763485.229478:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763485.229495:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494151168:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763485.932634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763486.956526:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763487.980569:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763489.004576:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763490.028553:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763490.348593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763490.348602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763490.348623:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763490.348633:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3494 00000100:00080000:0.0:1789763490.348645:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763490.348653:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763490.348718:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494151296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763490.348762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763490.348775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763490.348777:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763490.348782:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3494 00000100:00080000:0.0:1789763490.348786:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763490.348792:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763490.348803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763490.348819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494151296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763490.348820:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494151424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763490.348839:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763490.348868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763490.348872:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763490.348878:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3494 00000100:00080000:0.0:1789763490.348898:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763490.348903:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763490.348909:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763490.348924:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494151552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763490.348937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763490.348942:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763490.348944:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763490.348967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763490.348971:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3494 00000100:00080000:0.0:1789763490.348977:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763490.348982:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763490.348989:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763490.349001:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494151680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763490.349010:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763490] req@ffff8eb64304e300 x1876699494151296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763490.349052:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494151296/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763490.349063:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763490.349067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763490.349071:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494151296:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763490.349112:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763490.349132:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763490.349141:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763490] req@ffff8eb64304d180 x1876699494151424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763490.349177:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494151424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763490.349183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763490.349185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763490.349187:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151424:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763490.349191:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151552:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763490.349202:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763490.349208:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763490] req@ffff8eb64304c000 x1876699494151552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763490.349215:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494151552/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763490.349221:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763490.349222:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763490.349224:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151552:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763490.349247:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494151680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763490.349262:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763490.349272:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763490/real 1789763490] req@ffff8eb64304ed80 x1876699494151680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763490.349282:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494151680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763490.349290:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763490.349292:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763490.349295:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494151680:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763491.052508:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763492.076596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763493.100594:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763494.124634:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763495.148644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763495.468627:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.468640:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763495.468645:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.468658:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3499 00000100:00080000:2.0:1789763495.468672:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.468682:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763495.468729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494151808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.468779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.468784:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.468787:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.468792:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3499 00000100:00080000:2.0:1789763495.468796:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.468801:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763495.468853:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763495.468889:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494151808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763495.468900:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494151936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.468927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.468934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.468938:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.468946:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3499 00000100:00080000:2.0:1789763495.468955:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.468965:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763495.468976:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763495.469009:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763495.469009:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494152064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763495.469030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763495.469035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763495.469039:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763495.469046:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3499 00000100:00080000:2.0:1789763495.469053:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763495.469059:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763495.469068:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763495.469093:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff8eb650f80700 x1876699494151808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763495.469094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494152192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763495.469177:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494151808/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.469195:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.469201:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763495.469207:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494151808:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763495.469238:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.469282:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763495.469298:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff8eb650f80e00 x1876699494151936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.469322:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494151936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.469333:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.469337:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763495.469341:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494151936:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763495.469349:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152064:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763495.469369:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763495.469380:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff8eb650f83480 x1876699494152064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.469432:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494152064/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.469444:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.469448:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763495.469452:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152064:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763495.469460:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494152192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763495.469503:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763495.469524:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763495/real 1789763495] req@ffff8eb650f82a00 x1876699494152192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763495.469549:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494152192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763495.469596:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763495.469600:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763495.469605:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494152192:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763496.172625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763497.196516:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763498.220556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763499.244651:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763500.268577:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763500.588541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.588559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763500.588566:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.588584:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3505 00000100:00080000:0.0:1789763500.588602:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.588615:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763500.588670:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494152320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763500.588722:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.588731:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.588735:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.588741:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3505 00000100:00080000:0.0:1789763500.588748:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.588753:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763500.588789:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763500.588823:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494152448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763500.588845:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494152320:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763500.588853:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.588862:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.588866:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763500.588877:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3505 00000100:00080000:0.0:1789763500.588887:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.588897:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763500.588909:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763500.588933:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494152576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763500.588939:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763500.588987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763500.588994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763500.588998:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763500.589016:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff8eb64304c380 x1876699494152320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763500.589017:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3505 00000100:00080000:0.0:1789763500.589032:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763500.589051:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763500.589065:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763500.589088:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494152320/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763500.589090:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494152704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763500.589163:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763500.589171:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763500.589177:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494152320:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763500.589210:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763500.589246:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763500.589263:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff8eb64304dc00 x1876699494152448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763500.589281:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494152448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763500.589291:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763500.589296:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763500.589300:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152448:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763500.589308:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152576:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763500.589329:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763500.589340:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff8eb64304ce00 x1876699494152576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763500.589354:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494152576/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763500.589364:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763500.589368:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763500.589371:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152576:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763500.589378:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494152704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763500.589452:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763500.589470:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763500/real 1789763500] req@ffff8eb64304f800 x1876699494152704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763500.589487:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494152704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763500.589498:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763500.589502:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763500.589506:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494152704:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763501.292644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763502.316649:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763503.340677:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763504.364563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763505.388624:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763505.708706:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.708797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763505.708803:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.708828:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3510 00000100:00080000:1.0:1789763505.708850:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.708859:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763505.708932:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494152832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.708969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.708973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.708976:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.708981:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3510 00000100:00080000:1.0:1789763505.708985:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.709006:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763505.709019:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763505.709031:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494152832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763505.709042:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494152960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.709069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.709074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.709076:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.709081:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3510 00000100:00080000:1.0:1789763505.709085:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.709089:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763505.709094:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763505.709109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494153088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763505.709121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763505.709126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763505.709128:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763505.709132:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3510 00000100:00080000:1.0:1789763505.709139:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763505.709144:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763505.709149:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763505.709166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494153216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763505.709167:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763505.709230:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff8eb65016c700 x1876699494152832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763505.709254:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494152832/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763505.709267:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763505.709272:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763505.709277:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494152832:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763505.709328:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763505.709360:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763505.709376:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff8eb65016ce00 x1876699494152960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763505.709418:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494152960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763505.709429:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763505.709434:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763505.709438:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494152960:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763505.709446:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153088:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763505.709468:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763505.709481:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff8eb65016e300 x1876699494153088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763505.709524:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494153088/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763505.709534:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763505.709538:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763505.709541:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153088:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763505.709547:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494153216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763505.709568:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763505.709581:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763505/real 1789763505] req@ffff8eb65016c380 x1876699494153216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763505.709594:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494153216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763505.709602:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763505.709605:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763505.709609:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494153216:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763506.412616:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763507.436562:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763508.460546:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763509.484621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763510.508603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763510.828518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.828532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763510.828537:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.828551:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3515 00000100:00080000:1.0:1789763510.828565:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.828575:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763510.828622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494153344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.828664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.828669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.828672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.828678:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3515 00000100:00080000:1.0:1789763510.828683:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.828688:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763510.828724:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763510.828744:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494153472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.828761:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763510.828770:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494153344:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763510.828770:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.828779:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763510.828790:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3515 00000100:00080000:1.0:1789763510.828796:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.828800:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763510.828807:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763510.828828:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494153600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763510.828850:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763510.828856:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763510.828860:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763510.828866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763510.828878:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3515 00000100:00080000:1.0:1789763510.828889:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763510.828899:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763510.828911:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763510.828940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff8eb65016fb80 x1876699494153344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763510.828941:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494153728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763510.829011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494153344/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763510.829036:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763510.829043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763510.829048:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494153344:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763510.829091:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763510.829126:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763510.829144:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff8eb65016d500 x1876699494153472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763510.829161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494153472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763510.829173:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763510.829180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763510.829186:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153472:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763510.829199:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153600:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763510.829238:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763510.829261:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff8eb65016d880 x1876699494153600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763510.829286:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494153600/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763510.829311:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763510.829316:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763510.829320:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153600:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763510.829329:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494153728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763510.829359:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763510.829373:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763510/real 1789763510] req@ffff8eb65016ea00 x1876699494153728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763510.829388:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494153728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763510.829440:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763510.829445:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763510.829449:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494153728:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763511.532664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763512.556617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763513.580563:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763514.604550:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763515.628590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763515.948569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763515.948608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763515.948616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763515.948632:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3520 00000100:00080000:0.0:1789763515.948648:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763515.948690:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763515.948742:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494153856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763515.948782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763515.948787:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763515.948789:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763515.948795:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3520 00000100:00080000:0.0:1789763515.948800:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763515.948807:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763515.948820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763515.948838:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494153984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763515.948849:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494153856:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763515.948859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763515.948865:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763515.948867:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763515.948879:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3520 00000100:00080000:0.0:1789763515.948885:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763515.948889:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763515.948895:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763515.948912:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494154112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763515.948925:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763515.948930:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763515.948933:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763515.948939:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3520 00000100:00080000:0.0:1789763515.948955:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763515.948961:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763515.948966:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763515.948977:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763515.948986:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494154240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763515.949060:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763515/real 1789763515] req@ffff8eb64304d500 x1876699494153856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763515.949098:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494153856/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763515.949119:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763515.949127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763515.949136:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494153856:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763515.949173:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763515.949210:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763515.949228:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763515/real 1789763515] req@ffff8eb64304d880 x1876699494153984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763515.949247:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494153984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763515.949263:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763515.949271:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763515.949277:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494153984:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763515.949290:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154112:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763515.949333:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763515.949356:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763515/real 1789763515] req@ffff8eb64304f480 x1876699494154112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763515.949374:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494154112/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763515.949384:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763515.949388:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763515.949426:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154112:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763515.949435:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494154240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763515.949465:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763515.949481:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763515/real 1789763515] req@ffff8eb64304e680 x1876699494154240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763515.949496:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494154240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763515.949506:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763515.949510:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763515.949514:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494154240:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763516.652625:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763517.676584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763518.700582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763519.724589:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763520.748641:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763521.068541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.068554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763521.068559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.068572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3525 00000100:00080000:1.0:1789763521.068586:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.068595:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763521.068679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494154368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763521.068729:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.068733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.068736:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.068742:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3525 00000100:00080000:1.0:1789763521.068747:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.068755:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763521.068769:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763521.068794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494154368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763521.068794:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494154496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763521.068822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.068826:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.068829:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763521.068834:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3525 00000100:00080000:1.0:1789763521.068838:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.068842:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763521.068850:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763521.068867:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494154624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763521.068880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763521.068896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763521.068904:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763521.068905:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763521.068921:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3525 00000100:00080000:1.0:1789763521.068930:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763521.068935:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763521.068943:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763521.068959:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494154752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763521.069001:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff8eb65016e680 x1876699494154368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763521.069041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494154368/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763521.069058:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763521.069065:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763521.069071:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494154368:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763521.069107:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763521.069141:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763521.069159:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff8eb65016f800 x1876699494154496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763521.069175:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494154496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763521.069186:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763521.069190:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763521.069195:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154496:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763521.069202:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154624:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763521.069228:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763521.069247:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff8eb65016ea00 x1876699494154624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763521.069272:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494154624/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763521.069289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763521.069296:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763521.069302:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494154624:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763521.069314:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494154752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763521.069351:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763521.069367:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763521/real 1789763521] req@ffff8eb65016d880 x1876699494154752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763521.069381:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494154752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763521.069433:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763521.069438:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763521.069442:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494154752:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763521.772742:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763522.796680:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763523.820580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763524.844645:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763525.868621:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763526.188566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763526.188621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763526.188629:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763526.188647:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3530 00000100:00080000:3.0:1789763526.188662:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763526.188671:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763526.188720:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494154880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763526.188756:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763526.188762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763526.188764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763526.188770:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3530 00000100:00080000:3.0:1789763526.188774:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763526.188780:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763526.188793:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763526.188816:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494154880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763526.188817:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494155008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763526.188845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763526.188850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763526.188853:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763526.188859:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3530 00000100:00080000:3.0:1789763526.188864:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763526.188868:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763526.188877:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763526.188893:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494155136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763526.188907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763526.188914:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789763526.188918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763526.188920:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763526.188932:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3530 00000100:00080000:3.0:1789763526.188939:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763526.188945:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763526.188952:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763526.188967:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494155264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763526.188980:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff8eb64514bb80 x1876699494154880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763526.189007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494154880/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763526.189024:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763526.189031:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763526.189039:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494154880:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763526.189069:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763526.189095:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763526.189110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff8eb645148000 x1876699494155008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763526.189125:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494155008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763526.189135:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763526.189139:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763526.189143:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155008:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763526.189150:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155136:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763526.189170:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763526.189182:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff8eb64514b100 x1876699494155136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763526.189195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494155136/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763526.189205:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763526.189208:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763526.189228:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155136:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763526.189236:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494155264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763526.189257:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763526.189270:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763526/real 1789763526] req@ffff8eb645149880 x1876699494155264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763526.189284:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494155264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763526.189294:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763526.189297:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763526.189301:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494155264:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763526.892623:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763527.916637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763528.940620:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763529.964629:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763530.988609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763531.308649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763531.308665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763531.308672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763531.308688:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3535 00000100:00080000:0.0:1789763531.308707:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763531.308719:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763531.308784:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494155392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763531.308834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763531.308840:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763531.308842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763531.308848:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3535 00000100:00080000:0.0:1789763531.308853:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763531.308861:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763531.308877:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763531.308895:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494155520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763531.308917:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494155392:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763531.308919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763531.308942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763531.308947:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763531.308954:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3535 00000100:00080000:0.0:1789763531.308961:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763531.308967:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763531.308973:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763531.308990:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494155648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763531.309003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763531.309007:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763531.309010:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763531.309014:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3535 00000100:00080000:0.0:1789763531.309032:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763531.309038:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763531.309049:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763531.309053:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763531.309077:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494155776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763531.309132:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff8eb64304ea00 x1876699494155392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.309161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494155392/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.309177:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.309201:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763531.309206:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494155392:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763531.309242:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763531.309276:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763531.309293:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff8eb64304c700 x1876699494155520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.309309:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494155520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.309320:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.309325:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763531.309329:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155520:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763531.309338:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155648:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763531.309358:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763531.309383:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff8eb64304f100 x1876699494155648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.309443:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494155648/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.309454:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.309459:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763531.309463:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494155648:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763531.309472:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494155776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763531.309497:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763531.309511:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763531/real 1789763531] req@ffff8eb64304ca80 x1876699494155776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763531.309526:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494155776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763531.309537:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763531.309585:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763531.309592:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494155776:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763532.012582:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763533.036601:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763534.060617:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763535.084600:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763536.108616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763536.428592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.428607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763536.428621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.428636:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3540 00000100:00080000:3.0:1789763536.428652:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.428664:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763536.428713:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494155904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.428769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.428774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.428777:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.428782:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3540 00000100:00080000:3.0:1789763536.428788:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.428796:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763536.428809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763536.428819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494155904:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763536.428832:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494156032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.428871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.428875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.428877:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.428882:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3540 00000100:00080000:3.0:1789763536.428887:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.428890:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763536.428897:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763536.428911:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494156160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763536.428921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763536.428925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763536.428927:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763536.428931:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3540 00000100:00080000:3.0:1789763536.428936:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763536.428939:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763536.428944:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763536.428964:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494156288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763536.428975:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156032:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763536.429037:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156160:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763536.429102:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494156288:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763537.132706:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763538.156635:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763539.180651:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763539.500997:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763539.501079:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763539.501111:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763539.501207:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763539.501227:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763539.501241:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763539.501468:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763539] req@ffff8eb645148700 x1876699494156288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.501541:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494156288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.501563:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.501573:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763539.501582:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494156288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763539.501599:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763539] req@ffff8eb645149180 x1876699494156032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.501625:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494156032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.501665:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.501672:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763539.501680:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156032:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763539.501693:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763539] req@ffff8eb64514ad80 x1876699494156160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.501719:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494156160/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.501733:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.501739:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763539.501746:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156160:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763539.501759:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763536/real 1789763539] req@ffff8eb64514b480 x1876699494155904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.501783:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494155904/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.501798:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.501804:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763539.501811:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494155904:192.168.203.160@tcp:250:kworker.0 00010000:00010000:1.0:1789763540.204641:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763541.228720:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763541.548569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.548593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763541.548601:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.548624:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3546 00000100:00080000:3.0:1789763541.548647:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.548664:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763541.548740:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494156416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.548818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.548826:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.548829:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.548838:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3546 00000100:00080000:3.0:1789763541.548845:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.548853:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763541.548872:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763541.548901:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494156544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763541.548914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494156416:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763541.548931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.548940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.548944:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.548953:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3546 00000100:00080000:3.0:1789763541.548960:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.548968:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763541.548979:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763541.549006:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494156672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763541.549027:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763541.549034:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763541.549038:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763541.549046:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3546 00000100:00080000:3.0:1789763541.549054:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763541.549062:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763541.549073:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763541.549074:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763541.549106:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494156800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763541.549160:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff8eb64514b480 x1876699494156416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763541.549201:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494156416/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763541.549222:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763541.549231:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763541.549238:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494156416:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763541.549285:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763541.549344:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763541.549372:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff8eb64514ad80 x1876699494156544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763541.549456:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494156544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763541.549502:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763541.549511:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763541.549518:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156544:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763541.549530:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156672:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763541.549577:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763541.549602:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff8eb645149180 x1876699494156672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763541.549625:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494156672/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763541.549652:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763541.549660:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763541.549668:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494156672:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763541.549680:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494156800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763541.549725:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763541.549746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763541/real 1789763541] req@ffff8eb645148700 x1876699494156800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763541.549766:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494156800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763541.549780:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763541.549785:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763541.549793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494156800:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763542.252703:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763543.276678:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763544.300647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763545.324673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763546.348672:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763546.668518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.668531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763546.668537:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.668550:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3551 00000100:00080000:3.0:1789763546.668565:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.668579:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763546.668638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494156928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.668690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.668695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.668698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.668705:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3551 00000100:00080000:3.0:1789763546.668711:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.668719:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763546.668733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763546.668757:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494156928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763546.668758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494157056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.668785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.668790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.668792:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.668798:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3551 00000100:00080000:3.0:1789763546.668802:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.668807:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763546.668815:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763546.668832:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494157184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763546.668846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763546.668850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763546.668852:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763546.668864:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3551 00000800:00000100:1.0:1789763546.668865:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763546.668875:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763546.668885:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763546.668893:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763546.668910:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494157312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763546.668931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff8eb645149880 x1876699494156928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763546.668963:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494156928/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763546.668979:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763546.668985:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763546.668991:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494156928:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763546.669020:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763546.669047:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763546.669062:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff8eb64514b100 x1876699494157056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763546.669078:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494157056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763546.669094:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763546.669101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763546.669108:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157056:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763546.669120:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157184:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763546.669174:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763546.669198:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff8eb645148000 x1876699494157184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763546.669218:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494157184/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763546.669229:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763546.669233:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763546.669237:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157184:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763546.669245:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494157312:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763546.669273:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763546.669286:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763546/real 1789763546] req@ffff8eb64514bb80 x1876699494157312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763546.669301:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494157312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763546.669311:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763546.669315:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763546.669319:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494157312:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763547.372645:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763548.396543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763549.420540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763550.444632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763551.468653:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763551.788560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763551.788573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763551.788579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763551.788592:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3556 00000100:00080000:3.0:1789763551.788608:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763551.788617:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763551.788666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494157440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763551.788702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763551.788707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763551.788710:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763551.788718:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3556 00000100:00080000:3.0:1789763551.788724:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763551.788732:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763551.788749:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763551.788776:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494157568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763551.788788:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494157440:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763551.788816:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763551.788825:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763551.788829:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763551.788838:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3556 00000100:00080000:3.0:1789763551.788847:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763551.788855:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763551.788864:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763551.788884:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494157696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763551.788899:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763551.788909:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763551.788916:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763551.788920:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763551.788929:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3556 00000100:00080000:3.0:1789763551.788935:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763551.788941:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763551.788949:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763551.788958:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff8eb64514b800 x1876699494157440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763551.788969:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494157824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763551.789011:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494157440/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.789029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763551.789035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763551.789042:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494157440:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763551.789071:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157568:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763551.789098:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763551.789113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff8eb645148e00 x1876699494157568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763551.789138:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494157568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.789148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763551.789152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763551.789156:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763551.789164:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157696:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763551.789185:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763551.789203:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff8eb645148380 x1876699494157696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763551.789218:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494157696/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.789227:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763551.789231:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763551.789235:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494157696:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763551.789241:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494157824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763551.789271:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763551.789282:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763551/real 1789763551] req@ffff8eb64514a300 x1876699494157824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763551.789296:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494157824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763551.789314:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763551.789328:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763551.789332:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494157824:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763552.492685:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763553.516676:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763554.540645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763555.564632:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763556.588613:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763556.908497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763556.908508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763556.908513:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763556.908525:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3561 00000100:00080000:3.0:1789763556.908538:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763556.908546:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763556.908585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494157952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763556.908629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763556.908634:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763556.908636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763556.908640:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3561 00000100:00080000:3.0:1789763556.908655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763556.908659:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763556.908671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763556.908687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494158080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763556.908703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763556.908712:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494157952:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763556.908713:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763556.908719:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763556.908727:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3561 00000100:00080000:3.0:1789763556.908740:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763556.908751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763556.908760:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763556.908789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494158208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763556.908807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763556.908817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789763556.908818:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763556.908821:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763556.908830:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3561 00000100:00080000:3.0:1789763556.908838:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763556.908845:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763556.908853:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763556.908879:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763556/real 1789763556] req@ffff8eb645149500 x1876699494157952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763556.908880:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494158336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763556.908952:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494157952/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763556.908973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763556.908981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763556.908988:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494157952:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763556.909024:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763556.909066:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763556.909090:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763556/real 1789763556] req@ffff8eb645148a80 x1876699494158080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763556.909107:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494158080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763556.909117:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763556.909122:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763556.909126:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158080:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763556.909134:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158208:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763556.909165:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763556.909176:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763556/real 1789763556] req@ffff8eb64514a680 x1876699494158208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763556.909190:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494158208/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763556.909200:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763556.909203:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763556.909206:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158208:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763556.909212:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494158336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763556.909230:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763556.909251:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763556/real 1789763556] req@ffff8eb645149c00 x1876699494158336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763556.909264:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494158336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763556.909273:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763556.909276:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763556.909280:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494158336:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763557.612559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763558.636603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763559.660658:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763560.684612:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763561.708602:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763562.028561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763562.028577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763562.028585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763562.028603:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3566 00000100:00080000:3.0:1789763562.028623:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763562.028635:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763562.028705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494158464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763562.028756:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763562.028762:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763562.028765:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763562.028771:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3566 00000100:00080000:3.0:1789763562.028776:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763562.028787:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763562.028805:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763562.028831:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494158592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763562.028854:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763562.028862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763562.028883:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763562.028895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494158464:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763562.028900:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3566 00000100:00080000:3.0:1789763562.028915:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763562.028923:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763562.028932:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763562.028956:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494158720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763562.028982:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763562.028989:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763562.028993:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763562.029002:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3566 00000100:00080000:3.0:1789763562.029014:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763562.029023:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763562.029035:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763562.029037:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763562.029067:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494158848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763562.029132:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff8eb645149c00 x1876699494158464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763562.029175:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494158464/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763562.029198:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763562.029204:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763562.029210:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494158464:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763562.029221:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763562.029259:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763562.029277:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff8eb64514a680 x1876699494158592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763562.029293:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494158592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763562.029343:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763562.029349:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763562.029353:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158592:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763562.029430:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158720:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763562.029464:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763562.029480:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff8eb645148a80 x1876699494158720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763562.029498:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494158720/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763562.029509:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763562.029513:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763562.029517:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494158720:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763562.029524:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494158848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763562.029544:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763562.029555:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763562/real 1789763562] req@ffff8eb645149500 x1876699494158848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763562.029570:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494158848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763562.029580:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763562.029583:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763562.029586:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494158848:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763562.732610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763563.756658:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763564.780622:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763565.804632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763566.828633:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763567.148575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763567.148588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763567.148594:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763567.148608:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3571 00000100:00080000:0.0:1789763567.148625:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763567.148638:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763567.148697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494158976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763567.148744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763567.148749:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763567.148752:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763567.148758:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3571 00000100:00080000:0.0:1789763567.148764:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763567.148769:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763567.148782:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763567.148803:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494159104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763567.148820:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763567.148829:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763567.148831:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494158976:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763567.148833:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763567.148847:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3571 00000100:00080000:0.0:1789763567.148853:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763567.148856:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763567.148862:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763567.148879:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494159232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763567.148892:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763567.148897:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763567.148900:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763567.148906:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3571 00000100:00080000:0.0:1789763567.148915:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763567.148920:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763567.148926:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763567.148936:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763567.148943:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494159360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763567.149033:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff8eb64304ca80 x1876699494158976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763567.149062:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494158976/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763567.149078:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763567.149084:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763567.149089:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494158976:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763567.149118:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763567.149147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763567.149163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff8eb64304f100 x1876699494159104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763567.149178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494159104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763567.149197:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763567.149203:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763567.149210:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159104:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763567.149221:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159232:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763567.149258:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763567.149306:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff8eb64304c700 x1876699494159232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763567.149334:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494159232/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763567.149351:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763567.149355:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763567.149360:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159232:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763567.149369:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494159360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763567.149462:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763567.149485:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763567/real 1789763567] req@ffff8eb64304ea00 x1876699494159360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763567.149501:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494159360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763567.149511:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763567.149515:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763567.149519:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494159360:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763567.852600:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763568.876591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763569.900603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763570.924645:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763571.948603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763572.268547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.268560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763572.268565:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.268580:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3576 00000100:00080000:1.0:1789763572.268595:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.268606:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763572.268656:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494159488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.268697:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.268703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.268706:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.268711:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3576 00000100:00080000:1.0:1789763572.268717:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.268722:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763572.268734:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763572.268750:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494159616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.268765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.268772:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.268777:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763572.268787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494159488:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763572.268795:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3576 00000100:00080000:1.0:1789763572.268805:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.268809:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763572.268816:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763572.268833:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494159744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763572.268846:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763572.268851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763572.268854:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763572.268859:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3576 00000100:00080000:1.0:1789763572.268865:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763572.268870:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763572.268876:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763572.268894:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494159872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763572.268897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763572.268963:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff8eb65016d500 x1876699494159488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763572.268990:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494159488/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763572.269015:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763572.269021:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763572.269026:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494159488:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763572.269055:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763572.269082:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763572.269097:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff8eb65016fb80 x1876699494159616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763572.269113:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494159616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763572.269123:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763572.269127:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763572.269131:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159616:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763572.269138:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159744:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763572.269158:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763572.269169:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff8eb65016c380 x1876699494159744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763572.269183:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494159744/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763572.269201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763572.269205:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763572.269209:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494159744:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763572.269215:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494159872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763572.269233:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763572.269276:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763572/real 1789763572] req@ffff8eb65016e300 x1876699494159872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763572.269291:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494159872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763572.269301:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763572.269305:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763572.269309:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494159872:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763572.972571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763573.996595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763575.020605:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763576.044601:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763577.068621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763577.388541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763577.388554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763577.388560:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763577.388573:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3581 00000100:00080000:0.0:1789763577.388605:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763577.388614:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763577.388653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494160000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763577.388693:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763577.388700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763577.388703:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763577.388711:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3581 00000100:00080000:0.0:1789763577.388720:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763577.388746:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763577.388763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763577.388768:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494160000:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763577.388797:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494160128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763577.388818:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763577.388823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763577.388826:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763577.388831:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3581 00000100:00080000:0.0:1789763577.388840:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763577.388846:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763577.388854:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763577.388880:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763577.388893:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494160256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763577.388908:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763577.388912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763577.388913:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763577.388917:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3581 00000100:00080000:0.0:1789763577.388922:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763577.388927:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763577.388954:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763577.388966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff8eb64304e680 x1876699494160000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763577.388985:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494160384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763577.389010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494160000/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763577.389028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763577.389034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763577.389043:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494160000:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763577.389075:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763577.389110:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763577.389128:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff8eb64304f480 x1876699494160128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763577.389144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494160128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763577.389155:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763577.389160:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763577.389165:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160128:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763577.389172:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160256:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763577.389238:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763577.389252:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff8eb64304d880 x1876699494160256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763577.389274:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494160256/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763577.389287:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763577.389293:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763577.389299:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160256:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763577.389309:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494160384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763577.389344:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763577.389360:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763577/real 1789763577] req@ffff8eb64304d500 x1876699494160384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763577.389373:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494160384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763577.389382:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763577.389386:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763577.389389:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494160384:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763578.092656:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763579.116625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763580.140607:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763581.164624:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763582.188673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763582.508583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763582.508598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763582.508604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763582.508618:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3587 00000100:00080000:1.0:1789763582.508633:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763582.508644:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763582.508693:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494160512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763582.508737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763582.508742:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763582.508745:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763582.508750:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3587 00000100:00080000:1.0:1789763582.508755:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763582.508759:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763582.508782:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763582.508797:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494160640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763582.508818:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763582.508826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763582.508830:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763582.508839:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3587 00000100:00080000:1.0:1789763582.508848:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763582.508856:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763582.508867:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763582.508869:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494160512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763582.508899:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494160768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763582.508921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763582.508927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763582.508930:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763582.508936:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3587 00000100:00080000:1.0:1789763582.508942:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763582.508948:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763582.508956:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763582.508964:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763582.508982:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494160896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763582.509038:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff8eb65016ce00 x1876699494160512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763582.509066:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494160512/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763582.509081:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763582.509088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763582.509093:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494160512:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763582.509103:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763582.509130:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763582.509144:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff8eb65016c700 x1876699494160640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763582.509159:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494160640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763582.509198:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763582.509202:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763582.509208:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160640:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763582.509240:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160768:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763582.509269:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763582.509283:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff8eb65016df80 x1876699494160768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763582.509298:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494160768/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763582.509308:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763582.509312:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763582.509316:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494160768:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763582.509323:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494160896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763582.509340:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763582.509351:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763582/real 1789763582] req@ffff8eb65016dc00 x1876699494160896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763582.509365:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494160896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763582.509376:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763582.509382:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763582.509388:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494160896:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763583.212620:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763584.236689:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763585.260561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763586.284607:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763587.308676:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763587.628521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.628534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763587.628539:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.628553:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3592 00000100:00080000:1.0:1789763587.628567:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.628576:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763587.628632:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494161024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.628678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.628686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.628689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.628698:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3592 00000100:00080000:1.0:1789763587.628707:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.628717:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763587.628737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763587.628752:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494161024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763587.628769:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494161152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.628788:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763587.628792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.628795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.628801:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3592 00000100:00080000:1.0:1789763587.628806:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.628812:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763587.628820:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763587.628837:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494161280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763587.628851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763587.628861:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763587.628862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763587.628870:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763587.628879:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3592 00000100:00080000:1.0:1789763587.628885:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763587.628891:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763587.628908:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763587.628924:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494161408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763587.628946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff8eb65016f480 x1876699494161024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763587.628980:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494161024/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763587.629011:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763587.629020:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763587.629029:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494161024:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763587.629071:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763587.629109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763587.629157:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff8eb65016ed80 x1876699494161152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763587.629198:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494161152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763587.629217:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763587.629224:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763587.629231:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161152:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763587.629244:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161280:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763587.629278:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763587.629294:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff8eb65016d180 x1876699494161280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763587.629319:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494161280/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763587.629330:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763587.629334:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763587.629338:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161280:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763587.629345:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494161408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763587.629364:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763587.629375:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763587/real 1789763587] req@ffff8eb65016c000 x1876699494161408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763587.629389:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494161408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763587.629435:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763587.629439:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763587.629443:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494161408:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763588.332673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763589.356595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763590.380549:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763591.404559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763592.428656:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763592.748541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763592.748567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763592.748573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763592.748587:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3597 00000100:00080000:1.0:1789763592.748620:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763592.748640:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763592.748689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494161536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763592.748728:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763592.748732:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763592.748733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763592.748737:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3597 00000100:00080000:1.0:1789763592.748740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763592.748743:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763592.748753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763592.748763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494161664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763592.748783:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763592.748788:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763592.748791:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763592.748792:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494161536:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763592.748797:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3597 00000100:00080000:1.0:1789763592.748803:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763592.748807:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763592.748811:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763592.748825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494161792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763592.748833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763592.748837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763592.748838:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763592.748841:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3597 00000100:00080000:1.0:1789763592.748844:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763592.748858:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763592.748862:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763592.748869:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494161920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763592.748892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763592.748955:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff8eb65016ca80 x1876699494161536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763592.748975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494161536/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763592.748988:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763592.748994:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763592.748998:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494161536:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763592.749024:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763592.749042:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763592.749053:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff8eb65016f100 x1876699494161664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763592.749063:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494161664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763592.749070:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763592.749072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763592.749075:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161664:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763592.749079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161792:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763592.749117:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763592.749145:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff8eb65016c000 x1876699494161792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763592.749162:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494161792/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763592.749170:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763592.749173:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763592.749176:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494161792:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763592.749182:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494161920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763592.749200:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763592.749232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763592/real 1789763592] req@ffff8eb65016d180 x1876699494161920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763592.749242:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494161920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763592.749249:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763592.749251:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763592.749254:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494161920:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763593.452644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763594.476626:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763595.500665:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763596.524685:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763597.548624:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763597.868560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763597.868576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763597.868582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763597.868598:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3602 00000100:00080000:3.0:1789763597.868616:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763597.868628:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763597.868684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494162048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763597.868733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763597.868741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763597.868744:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763597.868753:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3602 00000100:00080000:3.0:1789763597.868761:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763597.868769:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763597.868784:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763597.868809:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494162176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763597.868843:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763597.868851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763597.868858:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494162048:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763597.868858:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763597.868878:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3602 00000100:00080000:3.0:1789763597.868886:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763597.868894:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763597.868907:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763597.868936:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494162304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763597.868957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763597.868965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763597.868970:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763597.868982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763597.868986:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3602 00000100:00080000:3.0:1789763597.869000:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763597.869010:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763597.869021:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763597.869074:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494162432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763597.869107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763597/real 1789763597] req@ffff8eb64514a300 x1876699494162048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763597.869148:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494162048/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763597.869169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763597.869177:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763597.869185:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494162048:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763597.869219:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763597.869258:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763597.869275:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763597/real 1789763597] req@ffff8eb645148380 x1876699494162176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763597.869292:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494162176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763597.869303:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763597.869307:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763597.869311:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162176:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763597.869318:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162304:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763597.869340:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763597.869351:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763597/real 1789763597] req@ffff8eb645148e00 x1876699494162304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763597.869366:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494162304/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763597.869375:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763597.869379:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763597.869383:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162304:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763597.869389:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494162432:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763597.869457:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763597.869472:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763597/real 1789763597] req@ffff8eb64514b800 x1876699494162432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763597.869496:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494162432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763597.869506:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763597.869510:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763597.869514:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494162432:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763598.572685:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763599.596689:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763600.620571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763601.644597:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763602.668623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763602.988558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763602.988570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763602.988576:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763602.988589:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3607 00000100:00080000:1.0:1789763602.988602:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763602.988613:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763602.988674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494162560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763602.988709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763602.988714:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763602.988718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763602.988723:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3607 00000100:00080000:1.0:1789763602.988728:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763602.988733:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763602.988745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763602.988761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494162688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763602.988775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763602.988780:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763602.988782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763602.988787:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3607 00000100:00080000:1.0:1789763602.988791:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763602.988795:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763602.988801:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763602.988810:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494162560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763602.988817:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494162816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763602.988840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763602.988844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763602.988846:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763602.988851:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3607 00000100:00080000:1.0:1789763602.988856:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763602.988860:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763602.988866:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763602.988883:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494162944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763602.988992:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162688:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763602.989125:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763602.989183:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494162944:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763603.692586:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763604.716594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763605.740657:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763606.060832:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789763606.060918:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763606.060944:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763606.061046:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763606.061066:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763606.061086:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763606.061109:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763602/real 1789763606] req@ffff8eb65016dc00 x1876699494162816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763606.061158:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494162816/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763606.061181:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763606.061189:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763606.061197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162816:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763606.061212:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763602/real 1789763606] req@ffff8eb65016df80 x1876699494162944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763606.061236:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494162944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763606.061250:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763606.061257:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763606.061263:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494162944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763606.061275:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763602/real 1789763606] req@ffff8eb65016ed80 x1876699494162560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763606.061298:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494162560/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763606.061313:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763606.061320:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763606.061326:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494162560:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763606.061338:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763602/real 1789763606] req@ffff8eb65016f480 x1876699494162688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763606.061359:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494162688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763606.061374:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763606.061380:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763606.061386:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494162688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763606.764603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763607.788674:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763608.108537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763608.108551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763608.108556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763608.108570:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3612 00000100:00080000:2.0:1789763608.108594:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763608.108604:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763608.108649:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494163072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763608.108683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763608.108687:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763608.108690:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763608.108695:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3612 00000100:00080000:2.0:1789763608.108700:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763608.108708:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763608.108722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763608.108743:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494163072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763608.108744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494163200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763608.108772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763608.108776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763608.108778:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763608.108784:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3612 00000100:00080000:2.0:1789763608.108788:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763608.108792:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763608.108799:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763608.108821:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494163328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763608.108850:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763608.108852:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763608.108862:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763608.108866:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763608.108878:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3612 00000100:00080000:2.0:1789763608.108887:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763608.108896:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763608.108907:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763608.108918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff8eb650f81180 x1876699494163072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763608.108939:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494163456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763608.109007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494163072/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.109033:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.109043:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763608.109051:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494163072:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763608.109091:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763608.109151:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763608.109176:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff8eb650f83b80 x1876699494163200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.109203:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494163200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.109221:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.109229:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763608.109236:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163200:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763608.109250:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163328:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763608.109289:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763608.109314:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff8eb650f81f80 x1876699494163328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.109340:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494163328/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.109357:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.109363:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763608.109370:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163328:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763608.109383:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494163456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763608.109463:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763608.109485:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763608/real 1789763608] req@ffff8eb650f83800 x1876699494163456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763608.109509:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494163456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763608.109537:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763608.109544:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763608.109550:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494163456:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763608.812607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763609.836640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763610.860582:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763611.884623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763612.908588:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763613.228552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763613.228563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763613.228568:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763613.228580:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3617 00000100:00080000:3.0:1789763613.228604:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763613.228613:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763613.228654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494163584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763613.228686:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763613.228690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763613.228692:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763613.228717:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3617 00000100:00080000:3.0:1789763613.228721:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763613.228726:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763613.228738:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763613.228755:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494163712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763613.228771:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763613.228776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763613.228778:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763613.228786:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494163584:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763613.228788:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3617 00000100:00080000:3.0:1789763613.228798:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763613.228807:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763613.228814:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763613.228834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494163840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763613.228849:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763613.228863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763613.228866:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763613.228872:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3617 00000100:00080000:3.0:1789763613.228878:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763613.228885:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763613.228894:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763613.228899:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763613.228921:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494163968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763613.229003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff8eb64514bb80 x1876699494163584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.229029:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494163584/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.229043:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.229048:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763613.229052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494163584:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763613.229079:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763613.229106:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763613.229120:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff8eb645148000 x1876699494163712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.229134:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494163712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.229144:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.229147:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763613.229152:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163712:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763613.229158:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163840:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763613.229176:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763613.229185:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff8eb64514b100 x1876699494163840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.229197:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494163840/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.229206:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.229209:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763613.229212:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494163840:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763613.229218:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494163968:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763613.229252:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763613.229264:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763613/real 1789763613] req@ffff8eb645149880 x1876699494163968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763613.229277:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494163968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763613.229285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763613.229288:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763613.229292:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494163968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763613.932628:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763614.956596:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763615.980669:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763617.004618:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763618.028577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763618.348510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763618.348521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763618.348526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763618.348539:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3622 00000100:00080000:3.0:1789763618.348552:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763618.348560:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763618.348617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494164096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763618.348652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763618.348657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763618.348660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763618.348675:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3622 00000100:00080000:3.0:1789763618.348680:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763618.348685:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763618.348698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763618.348715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494164224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763618.348731:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763618.348737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763618.348747:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494164096:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763618.348749:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763618.348764:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3622 00000100:00080000:3.0:1789763618.348769:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763618.348772:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763618.348779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763618.348797:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494164352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763618.348810:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763618.348815:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763618.348818:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763618.348822:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3622 00000100:00080000:3.0:1789763618.348827:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763618.348832:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763618.348838:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763618.348857:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494164480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763618.348862:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763618.348970:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff8eb645148700 x1876699494164096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.349010:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494164096/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.349025:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.349030:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763618.349035:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494164096:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763618.349064:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164224:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.349097:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763618.349113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff8eb645149180 x1876699494164224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.349129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494164224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.349140:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.349145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763618.349149:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763618.349157:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164352:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763618.349176:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763618.349187:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff8eb64514ad80 x1876699494164352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.349211:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494164352/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.349221:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.349224:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763618.349228:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164352:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763618.349234:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494164480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763618.349252:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763618.349263:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763618/real 1789763618] req@ffff8eb64514b480 x1876699494164480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763618.349277:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494164480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763618.349287:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763618.349290:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763618.349293:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494164480:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763619.052582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763620.076585:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763621.100628:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763622.124609:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763623.148620:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763623.468585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.468603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763623.468610:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.468627:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3627 00000100:00080000:2.0:1789763623.468646:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.468659:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763623.468736:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494164608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.468789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.468794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.468797:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.468804:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3627 00000100:00080000:2.0:1789763623.468810:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.468816:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763623.468834:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763623.468860:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494164736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763623.468911:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494164608:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763623.468918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.468937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.468941:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763623.468949:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3627 00000100:00080000:2.0:1789763623.468957:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.468965:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763623.468975:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763623.468999:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494164864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763623.469015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763623.469022:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763623.469025:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763623.469035:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:2.0:1789763623.469043:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3627 00000100:00080000:2.0:1789763623.469054:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763623.469064:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763623.469073:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763623.469094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494164992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763623.469110:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763623] req@ffff8eb650f80a80 x1876699494164608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763623.469143:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494164608/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763623.469160:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763623.469167:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763623.469172:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494164608:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763623.469204:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763623.469232:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763623.469248:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763623] req@ffff8eb650f80000 x1876699494164736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763623.469264:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494164736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763623.469275:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763623.469279:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763623.469284:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164736:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763623.469291:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164864:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763623.469311:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763623.469322:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763623] req@ffff8eb650f80380 x1876699494164864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763623.469337:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494164864/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763623.469347:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763623.469351:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763623.469354:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494164864:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763623.469363:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494164992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763623.469382:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763623.469444:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763623/real 1789763623] req@ffff8eb650f81500 x1876699494164992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763623.469461:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494164992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763623.469471:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763623.469476:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763623.469480:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494164992:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763624.172612:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763625.196636:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763626.220705:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763627.244682:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763628.268683:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763628.588592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763628.588604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763628.588609:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763628.588621:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3633 00000100:00080000:3.0:1789763628.588634:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763628.588644:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763628.588691:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494165120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763628.588729:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763628.588735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763628.588739:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763628.588747:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3633 00000100:00080000:3.0:1789763628.588755:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763628.588765:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763628.588782:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763628.588801:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494165120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763628.588816:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494165248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763628.588867:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763628.588873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763628.588877:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763628.588883:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3633 00000100:00080000:3.0:1789763628.588891:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763628.588897:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763628.588904:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763628.588907:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763628.588932:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494165376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763628.588949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763628.588954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763628.588957:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763628.588964:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff8eb64514b480 x1876699494165120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763628.588967:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3633 00000100:00080000:3.0:1789763628.588977:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763628.588990:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763628.589002:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763628.589013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494165120/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763628.589022:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494165504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763628.589050:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.589061:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763628.589069:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494165120:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763628.589102:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763628.589147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763628.589165:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff8eb64514ad80 x1876699494165248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.589182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494165248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.589199:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.589206:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763628.589213:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165248:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763628.589226:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165376:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763628.589267:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763628.589308:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff8eb645149180 x1876699494165376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.589336:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494165376/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.589354:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.589361:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763628.589369:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165376:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763628.589382:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494165504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763628.589486:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763628.589514:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763628/real 1789763628] req@ffff8eb645148700 x1876699494165504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763628.589531:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494165504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763628.589543:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763628.589547:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763628.589551:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494165504:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763629.292654:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763630.316600:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763631.340592:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763632.364595:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763633.388605:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763633.708508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763633.708522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763633.708528:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763633.708542:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3638 00000100:00080000:3.0:1789763633.708555:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763633.708564:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763633.708610:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494165632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763633.708644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763633.708648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763633.708651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763633.708657:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3638 00000100:00080000:3.0:1789763633.708661:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763633.708665:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763633.708676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763633.708692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494165760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763633.708706:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763633.708711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763633.708714:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763633.708719:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3638 00000100:00080000:3.0:1789763633.708722:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763633.708726:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763633.708732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763633.708746:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494165632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763633.708749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494165888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763633.708776:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763633.708780:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763633.708782:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763633.708809:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3638 00000100:00080000:3.0:1789763633.708815:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763633.708821:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763633.708829:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763633.708849:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494166016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763633.708890:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763633.708942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff8eb645149880 x1876699494165632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.708966:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494165632/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.708981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.708986:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763633.708992:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494165632:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763633.709002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.709027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763633.709041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff8eb64514b100 x1876699494165760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.709057:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494165760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.709068:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.709073:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763633.709077:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165760:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763633.709116:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165888:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763633.709154:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763633.709177:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff8eb645148000 x1876699494165888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.709202:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494165888/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.709220:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.709224:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763633.709229:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494165888:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763633.709237:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494166016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763633.709266:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763633.709280:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763633/real 1789763633] req@ffff8eb64514bb80 x1876699494166016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763633.709294:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494166016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763633.709305:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763633.709309:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763633.709314:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494166016:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763634.412582:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763635.436576:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763636.460641:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763637.484599:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763638.508653:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763638.828630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.828644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763638.828650:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.828690:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3643 00000100:00080000:0.0:1789763638.828708:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.828719:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763638.828810:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494166144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.828869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.828873:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.828876:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.828881:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3643 00000100:00080000:0.0:1789763638.828886:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.828890:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763638.828901:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.828919:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494166272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763638.828933:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.828938:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.828942:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.828947:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3643 00000100:00080000:0.0:1789763638.828951:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.828956:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763638.828967:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763638.828993:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494166400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763638.828993:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494166144:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763638.829046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763638.829053:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763638.829059:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763638.829070:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3643 00000100:00080000:0.0:1789763638.829079:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763638.829088:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763638.829101:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763638.829123:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494166528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763638.829140:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763638.829237:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff8eb64304f800 x1876699494166144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.829267:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494166144/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.829287:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.829294:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763638.829300:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494166144:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763638.829310:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.829339:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763638.829355:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff8eb64304ce00 x1876699494166272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.829370:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494166272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.829380:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.829385:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763638.829389:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166272:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763638.829491:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166400:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763638.829519:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763638.829533:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff8eb64304dc00 x1876699494166400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.829549:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494166400/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.829560:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.829564:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763638.829568:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166400:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763638.829586:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494166528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763638.829605:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763638.829617:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763638/real 1789763638] req@ffff8eb64304c380 x1876699494166528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763638.829631:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494166528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763638.829640:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763638.829643:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763638.829647:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494166528:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763639.532634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763640.556658:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763641.580659:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763642.604610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763643.628671:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763643.948523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763643.948547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763643.948552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763643.948564:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3648 00000100:00080000:0.0:1789763643.948578:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763643.948587:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763643.948637:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494166656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763643.948685:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763643.948693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763643.948697:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763643.948707:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3648 00000100:00080000:0.0:1789763643.948740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763643.948748:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763643.948775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763643.948787:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494166656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763643.948800:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494166784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763643.948829:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763643.948833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763643.948836:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763643.948841:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3648 00000100:00080000:0.0:1789763643.948848:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763643.948853:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763643.948860:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763643.948877:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494166912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763643.948893:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763643.948894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763643.948904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763643.948907:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763643.948916:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3648 00000100:00080000:0.0:1789763643.948923:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763643.948928:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763643.948934:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763643.948957:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763643/real 1789763643] req@ffff8eb64304ed80 x1876699494166656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763643.948960:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494167040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763643.949069:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494166656/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763643.949088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763643.949094:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763643.949100:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494166656:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763643.949131:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763643.949161:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763643.949178:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763643/real 1789763643] req@ffff8eb64304c000 x1876699494166784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763643.949195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494166784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763643.949212:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763643.949220:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763643.949227:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166784:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763643.949239:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166912:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763643.949281:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763643.949305:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763643/real 1789763643] req@ffff8eb64304d180 x1876699494166912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763643.949327:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494166912/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763643.949339:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763643.949344:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763643.949348:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494166912:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763643.949373:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494167040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763643.949444:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763643.949465:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763643/real 1789763643] req@ffff8eb64304e300 x1876699494167040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763643.949491:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494167040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763643.949507:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763643.949515:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763643.949530:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494167040:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763644.652619:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763645.676656:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763646.700604:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763647.724624:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763648.748650:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763649.068516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763649.068526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763649.068530:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763649.068541:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3653 00000100:00080000:3.0:1789763649.068553:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763649.068560:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763649.068629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494167168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763649.068671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763649.068687:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763649.068689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763649.068693:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3653 00000100:00080000:3.0:1789763649.068696:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763649.068700:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763649.068710:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763649.068723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494167296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763649.068732:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494167168:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763649.068734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763649.068741:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763649.068743:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763649.068746:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3653 00000100:00080000:3.0:1789763649.068748:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763649.068751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763649.068754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763649.068763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494167424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763649.068770:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763649.068773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763649.068774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763649.068777:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3653 00000100:00080000:3.0:1789763649.068779:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763649.068782:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763649.068786:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763649.068792:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494167552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763649.068810:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763649.068852:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff8eb64514b800 x1876699494167168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.068868:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494167168/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.068877:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.068880:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763649.068884:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494167168:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763649.068906:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763649.068922:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763649.068930:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff8eb645148e00 x1876699494167296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.068938:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494167296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.068944:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.068946:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763649.068949:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167296:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763649.068952:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167424:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763649.068963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763649.068968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff8eb645148380 x1876699494167424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.068975:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494167424/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.068981:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.068983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763649.068984:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167424:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763649.068987:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494167552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763649.068998:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763649.069004:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763649/real 1789763649] req@ffff8eb64514a300 x1876699494167552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763649.069015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494167552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763649.069020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763649.069022:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763649.069024:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494167552:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763649.772612:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763650.796642:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763651.820643:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763652.844690:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763653.868614:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763654.188529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.188542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763654.188547:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.188559:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3658 00000100:00080000:2.0:1789763654.188573:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.188583:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763654.188629:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494167680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.188690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.188695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763654.188697:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.188703:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3658 00000100:00080000:2.0:1789763654.188708:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.188713:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763654.188726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763654.188744:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494167808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.188758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.188766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763654.188768:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494167680:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763654.188769:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.188783:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3658 00000100:00080000:2.0:1789763654.188788:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.188792:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763654.188797:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763654.188813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494167936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763654.188825:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763654.188830:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763654.188832:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763654.188837:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3658 00000100:00080000:2.0:1789763654.188843:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763654.188847:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763654.188852:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763654.188866:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763654.188869:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494168064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763654.188926:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff8eb650f81500 x1876699494167680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.188951:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494167680/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.188965:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.188970:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763654.188975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494167680:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763654.189002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.189027:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763654.189041:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff8eb650f80380 x1876699494167808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.189056:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494167808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.189066:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.189072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763654.189078:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167808:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763654.189106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167936:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763654.189147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763654.189168:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff8eb650f80000 x1876699494167936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.189190:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494167936/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.189201:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.189205:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763654.189209:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494167936:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763654.189216:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494168064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763654.189243:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763654.189256:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763654/real 1789763654] req@ffff8eb650f80a80 x1876699494168064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763654.189270:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494168064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763654.189294:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763654.189298:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763654.189303:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494168064:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763654.892696:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763655.916593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763656.940570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763657.964644:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763658.988640:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763659.308563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763659.308576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763659.308582:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763659.308595:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3663 00000100:00080000:2.0:1789763659.308660:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763659.308670:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763659.308720:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494168192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763659.308765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763659.308770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763659.308773:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763659.308779:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3663 00000100:00080000:2.0:1789763659.308783:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763659.308787:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763659.308799:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763659.308815:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494168320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763659.308829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763659.308834:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763659.308837:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763659.308841:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3663 00000100:00080000:2.0:1789763659.308846:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763659.308857:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763659.308861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494168192:192.168.203.160@tcp:250:kworker.0 00000008:00100000:2.0:1789763659.308865:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763659.308884:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494168448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763659.308907:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763659.308912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763659.308915:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763659.308920:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3663 00000100:00080000:2.0:1789763659.308925:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763659.308929:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763659.308935:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763659.308954:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763659.308956:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494168576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763659.309044:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff8eb650f83800 x1876699494168192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.309071:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494168192/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.309088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.309095:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763659.309102:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494168192:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763659.309118:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.309147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763659.309162:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff8eb650f81f80 x1876699494168320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.309178:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494168320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.309189:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.309193:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763659.309197:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168320:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763659.309227:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168448:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763659.309252:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763659.309265:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff8eb650f83b80 x1876699494168448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.309279:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494168448/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.309289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.309294:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763659.309298:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168448:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763659.309304:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494168576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763659.309322:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763659.309333:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763659/real 1789763659] req@ffff8eb650f81180 x1876699494168576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763659.309346:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494168576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763659.309356:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763659.309360:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763659.309363:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494168576:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763660.012701:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763661.036682:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763662.060660:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763663.084725:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763664.108662:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763664.428515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763664.428528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763664.428534:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763664.428547:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3668 00000100:00080000:2.0:1789763664.428584:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763664.428596:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763664.428643:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494168704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763664.428680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763664.428685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763664.428688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763664.428693:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3668 00000100:00080000:2.0:1789763664.428698:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763664.428702:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763664.428722:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763664.428740:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494168832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763664.428755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763664.428760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763664.428763:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763664.428768:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3668 00000100:00080000:2.0:1789763664.428772:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763664.428777:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763664.428784:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763664.428802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494168704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763664.428803:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494168960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763664.428832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763664.428837:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763664.428839:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763664.428843:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3668 00000100:00080000:2.0:1789763664.428848:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763664.428852:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763664.428857:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763664.428872:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494169088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763664.428918:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763664.428997:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff8eb650f82a00 x1876699494168704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.429035:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494168704/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.429054:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.429062:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763664.429070:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494168704:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763664.429084:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.429126:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763664.429151:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff8eb650f83480 x1876699494168832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.429175:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494168832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.429191:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.429197:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763664.429203:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168832:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763664.429239:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168960:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763664.429273:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763664.429293:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff8eb650f80e00 x1876699494168960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.429317:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494168960/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.429342:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.429348:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763664.429354:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494168960:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763664.429364:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494169088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763664.429446:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763664.429473:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763664/real 1789763664] req@ffff8eb650f80700 x1876699494169088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763664.429498:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494169088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763664.429514:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763664.429521:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763664.429528:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494169088:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763665.132663:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763666.156660:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763667.180618:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763668.204615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763669.228702:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763669.548570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763669.548587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763669.548594:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763669.548611:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3674 00000100:00080000:2.0:1789763669.548628:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763669.548640:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763669.548693:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494169216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763669.548733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763669.548738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763669.548740:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763669.548746:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3674 00000100:00080000:2.0:1789763669.548751:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763669.548756:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763669.548770:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763669.548787:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494169344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763669.548800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763669.548805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763669.548808:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763669.548812:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3674 00000100:00080000:2.0:1789763669.548816:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763669.548820:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763669.548826:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763669.548847:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494169472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763669.548849:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494169216:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763669.548900:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763669.548904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763669.548906:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763669.548914:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3674 00000100:00080000:2.0:1789763669.548917:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763669.548922:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763669.548927:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763669.548940:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494169600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763669.549117:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169344:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763669.549186:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763669.549219:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494169600:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763670.252732:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763671.276665:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763672.300624:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763672.620721:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763672.620769:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763672.620787:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763672.620827:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763672.620836:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763672.620844:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763672.620908:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763672] req@ffff8eb650f82d80 x1876699494169472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763672.620949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494169472/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763672.620968:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763672.620975:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763672.620982:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169472:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763672.620993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763672] req@ffff8eb650f81c00 x1876699494169600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763672.621015:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494169600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763672.621028:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763672.621034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763672.621040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494169600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763672.621051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763672] req@ffff8eb650f81880 x1876699494169216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763672.621072:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494169216/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763672.621086:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763672.621091:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763672.621096:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494169216:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763672.621107:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763669/real 1789763672] req@ffff8eb650f82680 x1876699494169344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763672.621126:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494169344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763672.621138:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763672.621143:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763672.621149:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763673.324601:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763674.348651:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763674.668632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.668649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763674.668656:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.668672:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3679 00000100:00080000:2.0:1789763674.668689:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.668702:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763674.668757:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82680 x1876699494169728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.668828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.668836:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.668840:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.668849:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3679 00000100:00080000:2.0:1789763674.668857:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.668866:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763674.668883:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763674.668906:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81880 x1876699494169856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.668921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.668926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.668929:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.668937:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3679 00000100:00080000:2.0:1789763674.668942:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.668946:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763674.668953:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763674.668965:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81c00 x1876699494169984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763674.668979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763674.668984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763674.668988:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763674.668997:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3679 00000100:00080000:2.0:1789763674.669005:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763674.669015:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763674.669034:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763674.669035:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494169728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763674.669068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82d80 x1876699494170112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763674.669218:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763674.669328:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff8eb650f82680 x1876699494169728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.669372:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82680 x1876699494169728/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.669478:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.669527:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763674.669537:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494169728:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763674.669555:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.669609:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763674.669640:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff8eb650f81880 x1876699494169856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.669667:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81880 x1876699494169856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.669697:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.669704:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763674.669710:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169856:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763674.669725:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169984:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763674.669772:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763674.669796:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff8eb650f81c00 x1876699494169984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.669818:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81c00 x1876699494169984/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.669834:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.669844:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763674.669852:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494169984:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763674.669914:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494170112:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763674.669961:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763674.669987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763674/real 1789763674] req@ffff8eb650f82d80 x1876699494170112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763674.670012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82d80 x1876699494170112/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763674.670027:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763674.670034:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763674.670062:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494170112:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763675.372656:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763676.396581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763677.420643:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763678.444659:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763679.468610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763679.788650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.788663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763679.788669:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.788682:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3684 00000100:00080000:0.0:1789763679.788696:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.788707:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763679.788766:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494170240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.788799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.788804:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.788807:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.788812:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3684 00000100:00080000:0.0:1789763679.788816:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.788821:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763679.788832:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.788846:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494170368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.788858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763679.788862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.788864:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.788870:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3684 00000100:00080000:0.0:1789763679.788875:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.788883:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763679.788893:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763679.788916:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494170496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763679.788941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763679.788945:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494170240:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763679.788956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763679.788964:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763679.788974:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3684 00000100:00080000:0.0:1789763679.788982:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763679.788992:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763679.789003:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763679.789029:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494170624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763679.789044:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763679.789128:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff8eb64304e300 x1876699494170240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763679.789155:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494170240/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763679.789174:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763679.789180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763679.789187:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494170240:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763679.789198:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170368:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763679.789236:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763679.789260:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff8eb64304d180 x1876699494170368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763679.789285:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494170368/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763679.789303:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763679.789309:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763679.789317:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170368:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763679.789359:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170496:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763679.789442:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763679.789491:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff8eb64304c000 x1876699494170496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763679.789508:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494170496/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763679.789520:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763679.789524:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763679.789528:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170496:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763679.789536:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494170624:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763679.789564:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763679.789577:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763679/real 1789763679] req@ffff8eb64304ed80 x1876699494170624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763679.789591:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494170624/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763679.789601:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763679.789605:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763679.789609:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494170624:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763680.492644:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763681.516627:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763682.540674:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763683.564664:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763684.588645:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763684.908555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763684.908568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763684.908573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763684.908588:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3689 00000100:00080000:0.0:1789763684.908602:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763684.908613:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763684.908662:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494170752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763684.908693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763684.908698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763684.908701:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763684.908706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3689 00000100:00080000:0.0:1789763684.908710:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763684.908715:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763684.908726:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763684.908740:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494170880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763684.908753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763684.908757:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763684.908759:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763684.908763:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3689 00000100:00080000:0.0:1789763684.908767:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763684.908772:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763684.908777:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763684.908789:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494171008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763684.908802:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763684.908806:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763684.908809:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763684.908818:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494170752:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763684.908820:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3689 00000100:00080000:0.0:1789763684.908832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763684.908838:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763684.908845:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763684.908872:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494171136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763684.908915:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763684.908966:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763684/real 1789763684] req@ffff8eb64304c380 x1876699494170752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763684.908991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494170752/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763684.909004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763684.909010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763684.909034:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494170752:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763684.909043:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170880:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763684.909070:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763684.909085:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763684/real 1789763684] req@ffff8eb64304dc00 x1876699494170880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763684.909101:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494170880/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763684.909111:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763684.909115:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763684.909119:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494170880:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763684.909150:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171008:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763684.909170:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763684.909181:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763684/real 1789763684] req@ffff8eb64304ce00 x1876699494171008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763684.909195:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494171008/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763684.909214:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763684.909218:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763684.909222:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171008:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763684.909228:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494171136:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763684.909246:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763684.909257:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763684/real 1789763684] req@ffff8eb64304f800 x1876699494171136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763684.909270:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494171136/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763684.909281:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763684.909284:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763684.909287:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494171136:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763685.612610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763686.636625:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763687.660592:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763688.684633:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763689.708601:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763690.028505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763690.028522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763690.028529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763690.028544:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3694 00000100:00080000:0.0:1789763690.028556:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763690.028566:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763690.028610:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494171264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763690.028660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763690.028668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763690.028672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763690.028680:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3694 00000100:00080000:0.0:1789763690.028688:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763690.028698:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763690.028715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763690.028740:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494171264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763690.028749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494171392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763690.028788:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763690.028793:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763690.028796:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763690.028801:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3694 00000100:00080000:0.0:1789763690.028819:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763690.028826:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763690.028835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763690.028839:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763690.028865:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494171520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763690.028880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763690.028886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763690.028889:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763690.028896:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763690] req@ffff8eb64304d500 x1876699494171264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763690.028901:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3694 00000100:00080000:0.0:1789763690.028913:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763690.028924:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763690.028938:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763690.028949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494171264/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763690.028960:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494171648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763690.028985:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763690.028995:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763690.029003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494171264:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763690.029037:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171392:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763690.029065:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763690.029080:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763690] req@ffff8eb64304d880 x1876699494171392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763690.029096:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494171392/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763690.029106:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763690.029110:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763690.029114:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171392:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763690.029121:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171520:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763690.029141:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763690.029152:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763690] req@ffff8eb64304f480 x1876699494171520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763690.029166:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494171520/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763690.029177:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763690.029180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763690.029183:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171520:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763690.029190:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494171648:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763690.029208:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763690.029220:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763690/real 1789763690] req@ffff8eb64304e680 x1876699494171648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763690.029234:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494171648/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763690.029243:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763690.029247:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763690.029250:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494171648:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763690.732639:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763691.756603:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763692.780645:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763693.804601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763694.828594:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763695.148574:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763695.148586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763695.148591:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763695.148605:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3699 00000100:00080000:2.0:1789763695.148619:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763695.148628:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763695.148672:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80700 x1876699494171776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763695.148710:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763695.148714:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763695.148717:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763695.148729:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3699 00000100:00080000:2.0:1789763695.148733:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763695.148738:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763695.148748:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763695.148763:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80e00 x1876699494171904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763695.148777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763695.148782:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763695.148785:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763695.148789:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3699 00000100:00080000:2.0:1789763695.148793:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763695.148797:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763695.148803:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763695.148813:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494171776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763695.148820:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83480 x1876699494172032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763695.148836:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763695.148840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763695.148842:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763695.148845:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3699 00000100:00080000:2.0:1789763695.148848:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763695.148852:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763695.148858:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763695.148870:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f82a00 x1876699494172160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763695.148936:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763695.149026:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff8eb650f80700 x1876699494171776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763695.149055:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80700 x1876699494171776/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763695.149069:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763695.149075:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763695.149080:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494171776:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763695.149089:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171904:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763695.149122:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763695.149137:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff8eb650f80e00 x1876699494171904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763695.149161:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80e00 x1876699494171904/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763695.149176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763695.149180:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763695.149184:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494171904:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763695.149211:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172032:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763695.149231:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763695.149249:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff8eb650f83480 x1876699494172032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763695.149263:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83480 x1876699494172032/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763695.149272:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763695.149275:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763695.149279:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172032:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763695.149285:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494172160:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763695.149302:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763695.149313:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763695/real 1789763695] req@ffff8eb650f82a00 x1876699494172160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763695.149325:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f82a00 x1876699494172160/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763695.149372:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763695.149377:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763695.149390:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494172160:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763695.852602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763696.876608:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763697.900608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763698.924606:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763699.948623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763700.268515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763700.268527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763700.268532:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763700.268545:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3704 00000100:00080000:2.0:1789763700.268558:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763700.268568:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763700.268614:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494172288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763700.268652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763700.268657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763700.268659:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763700.268665:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3704 00000100:00080000:2.0:1789763700.268668:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763700.268673:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763700.268696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763700.268714:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494172416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763700.268730:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763700.268735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763700.268737:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763700.268743:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3704 00000100:00080000:2.0:1789763700.268747:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763700.268752:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763700.268759:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763700.268764:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494172288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763700.268777:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494172544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763700.268793:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763700.268797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763700.268799:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763700.268803:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3704 00000100:00080000:2.0:1789763700.268809:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763700.268814:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763700.268820:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763700.268834:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494172672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763700.268859:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763700.268911:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff8eb650f81180 x1876699494172288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763700.268937:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494172288/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763700.268951:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763700.268957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763700.268962:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494172288:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763700.268972:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172416:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763700.269001:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763700.269024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff8eb650f83b80 x1876699494172416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763700.269049:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494172416/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763700.269067:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763700.269075:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763700.269082:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172416:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763700.269124:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172544:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763700.269170:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763700.269195:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff8eb650f81f80 x1876699494172544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763700.269213:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494172544/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763700.269224:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763700.269228:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763700.269233:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172544:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763700.269255:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494172672:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763700.269283:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763700.269326:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763700/real 1789763700] req@ffff8eb650f83800 x1876699494172672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763700.269345:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494172672/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763700.269362:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763700.269368:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763700.269375:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494172672:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763700.972581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763701.996571:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763703.020601:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763704.044649:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763705.068565:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763705.388477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.388486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763705.388490:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.388500:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3709 00000100:00080000:3.0:1789763705.388509:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.388517:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763705.388561:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494172800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.388599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.388603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.388605:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.388611:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3709 00000100:00080000:3.0:1789763705.388616:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.388622:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763705.388633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763705.388640:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494172800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763705.388654:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494172928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.388669:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.388673:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.388674:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.388678:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3709 00000100:00080000:3.0:1789763705.388682:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.388695:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763705.388703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763705.388735:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494173056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763705.388752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763705.388758:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763705.388762:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763705.388772:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3709 00000800:00000100:1.0:1789763705.388775:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763705.388780:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763705.388790:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763705.388798:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763705.388814:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494173184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763705.388841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff8eb645149500 x1876699494172800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763705.388875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494172800/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763705.388909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763705.388914:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763705.388919:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494172800:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763705.388948:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172928:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763705.388978:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763705.388994:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff8eb645148a80 x1876699494172928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763705.389009:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494172928/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763705.389016:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763705.389019:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763705.389022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494172928:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763705.389028:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173056:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763705.389045:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763705.389053:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff8eb64514a680 x1876699494173056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763705.389068:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494173056/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763705.389074:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763705.389076:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763705.389078:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173056:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763705.389082:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494173184:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763705.389096:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763705.389103:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763705/real 1789763705] req@ffff8eb645149c00 x1876699494173184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763705.389112:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494173184/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763705.389118:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763705.389120:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763705.389122:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494173184:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763706.092517:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763707.116647:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763708.140585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763709.164577:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763710.188612:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763710.508517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763710.508525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763710.508528:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763710.508536:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3715 00000100:00080000:2.0:1789763710.508546:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763710.508551:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763710.508584:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494173312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763710.508606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763710.508609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763710.508611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763710.508614:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3715 00000100:00080000:2.0:1789763710.508616:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763710.508618:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763710.508625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763710.508633:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494173440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763710.508640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763710.508642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763710.508644:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763710.508646:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3715 00000100:00080000:2.0:1789763710.508648:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763710.508650:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763710.508653:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763710.508659:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494173568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763710.508665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763710.508668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763710.508669:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763710.508672:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3715 00000100:00080000:2.0:1789763710.508674:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763710.508677:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763710.508681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763710.508687:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494173696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763710.508694:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494173312:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789763710.508749:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763710.508787:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff8eb650f80a80 x1876699494173312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763710.508803:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494173312/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.508812:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763710.508816:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763710.508820:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494173312:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763710.508825:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173440:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763710.508839:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763710.508846:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff8eb650f80000 x1876699494173440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763710.508855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494173440/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.508860:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763710.508863:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763710.508865:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173440:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763710.508868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173568:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763710.508878:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763710.508884:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff8eb650f80380 x1876699494173568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763710.508891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494173568/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.508897:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763710.508899:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763710.508900:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173568:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763710.508920:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494173696:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763710.508930:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763710.508936:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763710/real 1789763710] req@ffff8eb650f81500 x1876699494173696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763710.508943:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494173696/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763710.508948:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763710.508950:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763710.508952:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494173696:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763711.212590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763712.236597:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763713.260568:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763714.284611:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763715.308583:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763715.628506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763715.628515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763715.628519:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763715.628529:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3720 00000100:00080000:3.0:1789763715.628539:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763715.628545:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763715.628582:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494173824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763715.628607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763715.628610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763715.628611:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763715.628615:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3720 00000100:00080000:3.0:1789763715.628618:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763715.628621:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763715.628630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763715.628639:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494173952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763715.628647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763715.628650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763715.628651:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763715.628671:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3720 00000100:00080000:3.0:1789763715.628675:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763715.628679:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763715.628684:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763715.628696:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494174080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763715.628709:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763715.628713:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763715.628715:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763715.628721:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3720 00000100:00080000:3.0:1789763715.628725:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763715.628730:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763715.628736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763715.628748:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494174208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763715.628763:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494173824:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789763715.628877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763715.628940:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff8eb645149c00 x1876699494173824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763715.628968:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494173824/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763715.628984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763715.628990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763715.628995:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494173824:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763715.629005:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173952:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763715.629035:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763715.629049:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff8eb64514a680 x1876699494173952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763715.629065:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494173952/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763715.629075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763715.629079:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763715.629083:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494173952:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763715.629089:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174080:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763715.629109:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763715.629119:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff8eb645148a80 x1876699494174080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763715.629133:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494174080/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763715.629143:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763715.629147:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763715.629150:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174080:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763715.629180:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494174208:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763715.629255:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763715.629265:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763715/real 1789763715] req@ffff8eb645149500 x1876699494174208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763715.629280:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494174208/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763715.629287:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763715.629289:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763715.629291:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494174208:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763716.332685:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763717.356576:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763718.380665:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763719.404637:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763720.428574:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763720.748502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763720.748519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763720.748527:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763720.748543:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3725 00000100:00080000:3.0:1789763720.748561:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763720.748585:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763720.748640:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494174336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763720.748679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763720.748685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763720.748688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763720.748693:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3725 00000100:00080000:3.0:1789763720.748699:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763720.748704:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763720.748715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763720.748731:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494174464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763720.748744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763720.748748:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763720.748750:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763720.748756:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3725 00000100:00080000:3.0:1789763720.748761:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763720.748766:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763720.748775:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763720.748789:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494174592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763720.748806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763720.748811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763720.748815:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763720.748823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494174336:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763720.748824:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3725 00000100:00080000:3.0:1789763720.748834:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763720.748860:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763720.748866:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763720.748882:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494174720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763720.748970:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763720.749056:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff8eb64514a300 x1876699494174336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763720.749086:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494174336/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.749097:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763720.749101:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763720.749105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494174336:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763720.749113:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174464:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763720.749136:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763720.749169:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff8eb645148380 x1876699494174464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763720.749182:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494174464/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.749197:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763720.749201:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763720.749204:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174464:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763720.749229:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174592:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763720.749250:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763720.749260:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff8eb645148e00 x1876699494174592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763720.749271:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494174592/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.749279:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763720.749282:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763720.749285:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174592:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763720.749290:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494174720:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763720.749303:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763720.749311:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763720/real 1789763720] req@ffff8eb64514b800 x1876699494174720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763720.749321:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494174720/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763720.749328:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763720.749330:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763720.749334:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494174720:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763721.452612:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763722.476714:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763723.500593:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763724.524658:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763725.548655:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763725.868535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763725.868547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763725.868553:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763725.868566:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3730 00000100:00080000:3.0:1789763725.868581:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763725.868589:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763725.868636:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494174848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763725.868673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763725.868678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763725.868680:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763725.868686:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3730 00000100:00080000:3.0:1789763725.868691:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763725.868696:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763725.868707:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763725.868724:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494174976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763725.868748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763725.868753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763725.868755:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763725.868760:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3730 00000100:00080000:3.0:1789763725.868764:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763725.868773:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494174848:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763725.868774:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763725.868787:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763725.868799:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494175104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763725.868814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763725.868818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763725.868821:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763725.868827:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3730 00000100:00080000:3.0:1789763725.868832:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763725.868836:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763725.868842:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763725.868864:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494175232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763725.868883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763725.868977:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763725/real 1789763725] req@ffff8eb64514bb80 x1876699494174848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763725.869017:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494174848/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763725.869038:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763725.869046:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763725.869054:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494174848:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763725.869076:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174976:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763725.869134:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763725.869153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763725/real 1789763725] req@ffff8eb645148000 x1876699494174976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763725.869170:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494174976/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763725.869181:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763725.869185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763725.869190:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494174976:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763725.869220:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175104:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763725.869245:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763725.869258:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763725/real 1789763725] req@ffff8eb64514b100 x1876699494175104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763725.869272:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494175104/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763725.869282:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763725.869286:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763725.869290:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175104:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763725.869297:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494175232:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763725.869314:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763725.869325:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763725/real 1789763725] req@ffff8eb645149880 x1876699494175232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763725.869339:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494175232/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763725.869348:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763725.869351:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763725.869355:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494175232:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763726.572631:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763727.596626:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763728.620716:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763729.644607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763730.668594:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763730.988538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763730.988552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763730.988558:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763730.988571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3735 00000100:00080000:3.0:1789763730.988584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763730.988593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763730.988639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494175360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763730.988677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763730.988682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763730.988684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763730.988689:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3735 00000100:00080000:3.0:1789763730.988694:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763730.988698:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763730.988710:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763730.988725:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494175488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763730.988739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763730.988743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763730.988745:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763730.988750:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3735 00000100:00080000:3.0:1789763730.988753:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763730.988758:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763730.988766:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763730.988779:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494175616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763730.988794:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494175360:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763730.988798:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763730.988812:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763730.988815:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763730.988825:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3735 00000100:00080000:3.0:1789763730.988832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763730.988839:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763730.988849:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763730.988875:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494175744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763730.988891:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763730.988954:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763730/real 1789763730] req@ffff8eb645148700 x1876699494175360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763730.988982:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494175360/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763730.988996:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763730.989002:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763730.989008:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494175360:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763730.989017:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175488:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763730.989042:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763730.989057:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763730/real 1789763730] req@ffff8eb645149180 x1876699494175488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763730.989110:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494175488/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763730.989128:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763730.989135:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763730.989142:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175488:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763730.989182:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175616:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763730.989216:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763730.989232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763730/real 1789763730] req@ffff8eb64514ad80 x1876699494175616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763730.989247:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494175616/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763730.989266:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763730.989270:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763730.989275:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494175616:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763730.989282:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494175744:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763730.989301:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763730.989312:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763730/real 1789763730] req@ffff8eb64514b480 x1876699494175744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763730.989326:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494175744/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763730.989336:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763730.989339:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763730.989343:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494175744:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763731.692607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763732.716595:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763733.740570:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763734.764615:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763735.788585:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763736.108575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763736.108593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763736.108600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763736.108618:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3740 00000100:00080000:0.0:1789763736.108636:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763736.108648:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763736.108703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494175872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763736.108758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763736.108766:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763736.108770:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763736.108778:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3740 00000100:00080000:0.0:1789763736.108785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763736.108791:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763736.108806:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763736.108828:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494176000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763736.108846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763736.108856:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763736.108856:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494175872:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763736.108864:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763736.108879:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3740 00000100:00080000:0.0:1789763736.108882:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763736.108887:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763736.108895:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763736.108911:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494176128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763736.108926:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763736.108931:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763736.108933:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763736.108938:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3740 00000100:00080000:0.0:1789763736.108944:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763736.108948:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763736.108956:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763736.108978:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494176256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763736.109024:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763736.109106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763736.109135:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494176256:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763736.812646:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763737.836517:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763738.860584:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763739.180943:0:2356:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:3.0:1789763739.181060:0:2356:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763739.181094:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763739.181172:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763739.181193:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:3.0:1789763739.181213:0:2356:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763739.181283:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763739] req@ffff8eb64304c700 x1876699494176000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763739.181356:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494176000/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763739.181381:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763739.181392:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763739.181452:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176000:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763739.181470:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763739] req@ffff8eb64304f100 x1876699494176128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763739.181499:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494176128/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763739.181514:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763739.181521:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763739.181554:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176128:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763739.181572:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763739] req@ffff8eb64304ca80 x1876699494176256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763739.181609:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494176256/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763739.181624:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763739.181634:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763739.181643:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494176256:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763739.181660:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763736/real 1789763739] req@ffff8eb64304ea00 x1876699494175872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763739.181683:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494175872/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763739.181697:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763739.181703:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763739.181709:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494175872:192.168.203.160@tcp:250:kworker.0 00010000:00010000:1.0:1789763739.884632:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763740.908663:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763741.228601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763741.228613:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763741.228618:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763741.228645:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3745 00000100:00080000:3.0:1789763741.228659:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763741.228668:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763741.228713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494176384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763741.228759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763741.228765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763741.228767:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763741.228773:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3745 00000100:00080000:3.0:1789763741.228777:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763741.228782:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763741.228794:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763741.228811:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494176512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763741.228826:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763741.228832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763741.228834:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763741.228840:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3745 00000100:00080000:3.0:1789763741.228845:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763741.228849:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763741.228856:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763741.228865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494176384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763741.228880:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494176640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763741.228917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763741.228925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763741.228928:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763741.228938:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3745 00000100:00080000:3.0:1789763741.228946:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763741.228955:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763741.228967:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763741.228981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763741.229021:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494176768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763741.229082:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff8eb64514b480 x1876699494176384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763741.229118:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494176384/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763741.229138:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763741.229145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763741.229153:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494176384:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763741.229169:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176512:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763741.229210:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763741.229235:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff8eb64514ad80 x1876699494176512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763741.229262:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494176512/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763741.229291:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763741.229298:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763741.229306:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176512:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763741.229352:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176640:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763741.229466:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763741.229494:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff8eb645149180 x1876699494176640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763741.229520:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494176640/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763741.229538:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763741.229544:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763741.229552:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494176640:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763741.229568:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494176768:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763741.229611:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763741.229636:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763741/real 1789763741] req@ffff8eb645148700 x1876699494176768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763741.229676:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494176768/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763741.229694:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763741.229700:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763741.229707:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494176768:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763741.932607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763742.956603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763743.980672:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763745.004522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763746.028640:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763746.348485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763746.348496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763746.348500:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763746.348510:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3750 00000100:00080000:3.0:1789763746.348522:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763746.348530:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763746.348568:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494176896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763746.348599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763746.348602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763746.348604:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763746.348609:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3750 00000100:00080000:3.0:1789763746.348612:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763746.348616:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763746.348625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763746.348638:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494177024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763746.348654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763746.348661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763746.348665:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763746.348677:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3750 00000100:00100000:1.0:1789763746.348680:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494176896:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763746.348684:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763746.348697:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763746.348706:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763746.348729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494177152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763746.348748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763746.348754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763746.348758:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763746.348765:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3750 00000100:00080000:3.0:1789763746.348771:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763746.348778:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763746.348785:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763746.348796:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763746.348813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494177280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763746.348845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff8eb645149880 x1876699494176896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763746.348878:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494176896/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763746.348896:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763746.348902:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763746.348909:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494176896:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763746.348922:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177024:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763746.348983:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763746.349006:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff8eb64514b100 x1876699494177024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763746.349040:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494177024/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763746.349054:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763746.349060:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763746.349066:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177024:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763746.349101:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177152:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763746.349135:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763746.349154:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff8eb645148000 x1876699494177152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763746.349175:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494177152/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763746.349190:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763746.349196:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763746.349201:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177152:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763746.349211:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494177280:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763746.349240:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763746.349257:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763746/real 1789763746] req@ffff8eb64514bb80 x1876699494177280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763746.349275:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494177280/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763746.349289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763746.349294:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763746.349300:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494177280:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763747.052678:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763748.076673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763749.100625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763750.124541:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763751.148619:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763751.468529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.468541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763751.468546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.468557:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3755 00000100:00080000:0.0:1789763751.468569:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.468577:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763751.468638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494177408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.468679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.468685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.468689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.468694:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3755 00000100:00080000:0.0:1789763751.468699:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.468705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763751.468716:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763751.468734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494177536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763751.468743:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494177408:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763751.468753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.468760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.468762:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.468767:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3755 00000100:00080000:0.0:1789763751.468770:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.468774:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763751.468779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763751.468791:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494177664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763751.468802:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763751.468806:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763751.468809:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763751.468812:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3755 00000100:00080000:0.0:1789763751.468818:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763751.468822:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763751.468827:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763751.468840:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763751.468841:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494177792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763751.468904:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff8eb64304ea00 x1876699494177408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.468965:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494177408/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.468983:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.468990:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763751.468997:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494177408:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763751.469030:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177536:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.469059:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763751.469074:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff8eb64304ca80 x1876699494177536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.469088:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494177536/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.469098:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.469102:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763751.469105:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177536:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763751.469111:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177664:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763751.469129:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763751.469139:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff8eb64304f100 x1876699494177664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.469151:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494177664/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.469166:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.469169:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763751.469173:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494177664:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763751.469178:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494177792:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763751.469193:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763751.469202:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763751/real 1789763751] req@ffff8eb64304c700 x1876699494177792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763751.469226:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494177792/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763751.469235:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763751.469239:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763751.469242:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494177792:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763752.172611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763753.196569:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763754.220619:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763755.244627:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763756.268653:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763756.588573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763756.588607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763756.588612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763756.588624:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3761 00000100:00080000:3.0:1789763756.588636:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763756.588644:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763756.588687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494177920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763756.588719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763756.588723:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763756.588726:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763756.588730:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3761 00000100:00080000:3.0:1789763756.588734:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763756.588738:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763756.588748:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763756.588763:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494178048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763756.588776:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763756.588781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763756.588785:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763756.588789:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494177920:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763756.588794:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3761 00000100:00080000:3.0:1789763756.588806:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763756.588809:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763756.588815:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763756.588831:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494178176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763756.588842:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763756.588846:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763756.588849:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763756.588854:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3761 00000100:00080000:3.0:1789763756.588859:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763756.588863:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763756.588868:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763756.588880:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494178304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763756.588905:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763756.588982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763756] req@ffff8eb64514b800 x1876699494177920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763756.589007:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494177920/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763756.589020:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763756.589025:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763756.589030:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494177920:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763756.589057:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178048:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763756.589086:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763756.589099:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763756] req@ffff8eb645148e00 x1876699494178048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763756.589113:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494178048/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763756.589122:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763756.589125:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763756.589129:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178048:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763756.589135:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178176:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763756.589152:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763756.589162:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763756] req@ffff8eb645148380 x1876699494178176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763756.589174:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494178176/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763756.589183:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763756.589185:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763756.589189:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178176:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763756.589194:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494178304:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763756.589211:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763756.589221:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763756/real 1789763756] req@ffff8eb64514a300 x1876699494178304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763756.589233:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494178304/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763756.589241:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763756.589244:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763756.589247:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494178304:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763757.292636:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763758.316687:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763759.340610:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763760.364552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763761.388553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763761.708611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763761.708624:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763761.708631:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763761.708646:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3766 00000100:00080000:0.0:1789763761.708661:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763761.708671:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763761.708729:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494178432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763761.708776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763761.708780:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763761.708783:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763761.708787:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3766 00000100:00080000:0.0:1789763761.708791:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763761.708794:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763761.708808:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763761.708822:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494178560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763761.708835:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763761.708840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763761.708842:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763761.708860:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3766 00000100:00080000:0.0:1789763761.708864:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763761.708868:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763761.708874:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763761.708916:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494178688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763761.708930:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494178432:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763761.708960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763761.708964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763761.708967:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763761.708974:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3766 00000100:00080000:0.0:1789763761.708979:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763761.708984:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763761.708990:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763761.709002:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494178816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763761.709019:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763761.709060:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff8eb64304e680 x1876699494178432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763761.709075:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494178432/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763761.709083:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763761.709086:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763761.709090:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494178432:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763761.709096:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178560:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763761.709111:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763761.709120:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff8eb64304f480 x1876699494178560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763761.709129:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494178560/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763761.709135:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763761.709145:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763761.709148:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178560:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763761.709172:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178688:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763761.709195:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763761.709207:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff8eb64304d880 x1876699494178688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763761.709228:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494178688/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763761.709236:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763761.709239:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763761.709241:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494178688:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763761.709246:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494178816:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763761.709263:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763761.709271:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763761/real 1789763761] req@ffff8eb64304d500 x1876699494178816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763761.709279:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494178816/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763761.709289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763761.709292:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763761.709311:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494178816:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763762.412614:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763763.436647:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763764.460610:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763765.484658:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763766.508533:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763766.828636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763766.828652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763766.828659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763766.828674:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3771 00000100:00080000:3.0:1789763766.828691:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763766.828702:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763766.828761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494178944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763766.828803:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763766.828861:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763766.828863:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763766.828869:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3771 00000100:00080000:3.0:1789763766.828875:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763766.828881:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763766.828911:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763766.828931:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494179072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763766.828949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763766.828956:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494178944:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763766.828956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763766.828961:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763766.828975:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3771 00000100:00080000:3.0:1789763766.828981:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763766.828985:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763766.828992:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763766.829007:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494179200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763766.829022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763766.829027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763766.829029:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763766.829034:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3771 00000100:00080000:3.0:1789763766.829040:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763766.829045:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763766.829050:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763766.829063:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494179328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763766.829081:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763766.829153:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff8eb645149500 x1876699494178944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.829184:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494178944/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.829212:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.829218:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763766.829223:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494178944:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763766.829275:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179072:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.829303:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763766.829317:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff8eb645148a80 x1876699494179072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.829331:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494179072/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.829341:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.829344:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763766.829349:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179072:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763766.829355:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179200:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763766.829373:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763766.829383:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff8eb64514a680 x1876699494179200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.829394:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494179200/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.829403:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.829406:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763766.829466:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179200:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763766.829476:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494179328:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763766.829533:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763766.829547:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763766/real 1789763766] req@ffff8eb645149c00 x1876699494179328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763766.829579:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494179328/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763766.829590:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763766.829594:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763766.829598:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494179328:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763767.532562:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763768.556678:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763769.580638:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763770.604564:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763771.628675:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763771.948537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763771.948556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763771.948563:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763771.948603:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3776 00000100:00080000:0.0:1789763771.948622:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763771.948634:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763771.948697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494179456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763771.948736:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763771.948742:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763771.948745:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763771.948750:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3776 00000100:00080000:0.0:1789763771.948755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763771.948759:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763771.948797:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763771.948818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494179584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763771.948834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763771.948839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763771.948842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763771.948848:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3776 00000100:00080000:0.0:1789763771.948854:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763771.948860:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763771.948866:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763771.948881:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494179712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763771.948894:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494179456:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763771.948900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763771.948914:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763771.948917:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763771.948922:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3776 00000100:00080000:0.0:1789763771.948926:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763771.948931:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763771.948938:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763771.948954:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494179840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763771.949025:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763771.949085:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763771/real 1789763771] req@ffff8eb64304f800 x1876699494179456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763771.949112:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494179456/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763771.949127:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763771.949133:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763771.949139:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494179456:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763771.949148:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179584:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763771.949176:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763771.949190:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763771/real 1789763771] req@ffff8eb64304ce00 x1876699494179584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763771.949225:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494179584/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763771.949241:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763771.949248:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763771.949255:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179584:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763771.949304:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179712:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763771.949347:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763771.949364:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763771/real 1789763771] req@ffff8eb64304dc00 x1876699494179712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763771.949389:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494179712/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763771.949442:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763771.949446:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763771.949450:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494179712:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763771.949458:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494179840:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763771.949485:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763771.949499:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763771/real 1789763771] req@ffff8eb64304c380 x1876699494179840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763771.949514:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494179840/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763771.949523:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763771.949527:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763771.949531:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494179840:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763772.652623:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763773.676643:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763774.700565:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763775.724669:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763776.748814:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763777.068519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763777.068530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763777.068534:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763777.068544:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3781 00000100:00080000:3.0:1789763777.068556:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763777.068563:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763777.068601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494179968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763777.068629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763777.068632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763777.068633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763777.068637:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3781 00000100:00080000:3.0:1789763777.068640:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763777.068643:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763777.068651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763777.068661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494180096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763777.068669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763777.068672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763777.068673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763777.068676:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3781 00000100:00080000:3.0:1789763777.068678:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763777.068680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763777.068683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763777.068690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494180224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763777.068698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763777.068701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763777.068702:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763777.068706:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3781 00000100:00080000:3.0:1789763777.068709:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763777.068713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763777.068717:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763777.068724:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494180352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763777.068751:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494179968:192.168.203.160@tcp:250:kworker.0 00000800:00000100:1.0:1789763777.068826:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763777.068872:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff8eb645149c00 x1876699494179968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763777.068891:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494179968/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763777.068902:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763777.068906:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763777.068910:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494179968:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763777.068916:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180096:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763777.068937:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763777.068946:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff8eb64514a680 x1876699494180096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763777.068956:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494180096/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763777.068964:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763777.068967:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763777.068969:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180096:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763777.068974:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180224:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763777.068986:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763777.068993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff8eb645148a80 x1876699494180224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763777.069001:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494180224/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763777.069008:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763777.069010:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763777.069013:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180224:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763777.069034:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494180352:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763777.069047:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763777.069054:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763777/real 1789763777] req@ffff8eb645149500 x1876699494180352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763777.069063:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494180352/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763777.069070:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763777.069072:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763777.069074:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494180352:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763777.772535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763778.796560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763779.820556:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763780.844584:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763781.868493:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763782.188527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763782.188539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763782.188543:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763782.188553:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3786 00000100:00080000:1.0:1789763782.188573:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763782.188579:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763782.188613:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494180480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763782.188641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763782.188646:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763782.188648:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763782.188654:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3786 00000100:00080000:1.0:1789763782.188659:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763782.188665:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763782.188676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763782.188683:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494180480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763782.188706:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494180608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763782.188716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763782.188719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763782.188721:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763782.188724:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3786 00000100:00080000:1.0:1789763782.188739:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763782.188756:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789763782.188762:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:1.0:1789763782.188763:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763782.188781:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494180736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763782.188792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763782.188796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763782.188799:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff8eb65016f480 x1876699494180480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763782.188800:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763782.188812:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3786 00000100:00080000:1.0:1789763782.188818:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763782.188822:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494180480/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763782.188827:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763782.188837:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763782.188842:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789763782.188850:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494180864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763782.188851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763782.188860:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494180480:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763782.188891:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180608:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763782.188910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763782.188919:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff8eb65016ed80 x1876699494180608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.188927:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494180608/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.188943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.188946:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763782.188948:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180608:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763782.188952:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180736:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763782.188963:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763782.188969:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff8eb65016df80 x1876699494180736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.188976:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494180736/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.188982:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.188983:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763782.188985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494180736:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763782.188989:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494180864:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763782.189000:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763782.189006:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763782/real 1789763782] req@ffff8eb65016dc00 x1876699494180864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763782.189013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494180864/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763782.189018:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763782.189019:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763782.189022:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494180864:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763782.892553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763783.916514:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763784.940492:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763785.964568:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763786.988527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763787.308481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763787.308488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763787.308491:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763787.308499:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3791 00000100:00080000:3.0:1789763787.308508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763787.308514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763787.308550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494180992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763787.308571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763787.308573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763787.308575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763787.308578:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3791 00000100:00080000:3.0:1789763787.308583:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763787.308587:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763787.308594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763787.308603:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494181120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763787.308609:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494180992:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763787.308615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763787.308619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763787.308620:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763787.308623:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3791 00000100:00080000:3.0:1789763787.308625:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763787.308628:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763787.308631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763787.308640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494181248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763787.308647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763787.308650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763787.308651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763787.308653:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3791 00000100:00080000:3.0:1789763787.308673:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763787.308677:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763787.308681:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763787.308690:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:3.0:1789763787.308691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494181376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763787.308730:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff8eb64514a300 x1876699494180992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763787.308745:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494180992/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763787.308753:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763787.308756:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763787.308759:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494180992:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763787.308776:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181120:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763787.308792:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763787.308800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff8eb645148380 x1876699494181120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763787.308808:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494181120/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763787.308813:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763787.308815:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763787.308817:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181120:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763787.308823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181248:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763787.308833:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763787.308839:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff8eb645148e00 x1876699494181248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763787.308847:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494181248/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763787.308852:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763787.308854:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763787.308856:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181248:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763787.308859:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494181376:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763787.308868:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763787.308874:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763787/real 1789763787] req@ffff8eb64514b800 x1876699494181376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763787.308881:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494181376/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763787.308886:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763787.308887:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763787.308889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494181376:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763788.012504:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763789.036593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763790.060552:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763791.084522:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763792.108553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763792.428482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763792.428489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763792.428492:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763792.428500:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3796 00000100:00080000:3.0:1789763792.428508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763792.428514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763792.428547:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494181504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763792.428569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763792.428573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763792.428575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763792.428579:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3796 00000100:00080000:3.0:1789763792.428582:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763792.428587:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763792.428596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763792.428609:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494181632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763792.428637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763792.428641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763792.428643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763792.428644:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494181504:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763792.428658:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3796 00000100:00080000:3.0:1789763792.428661:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763792.428664:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763792.428669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763792.428679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494181760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763792.428688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763792.428691:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763792.428692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763792.428696:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3796 00000100:00080000:3.0:1789763792.428699:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763792.428702:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763792.428706:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763792.428715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494181888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763792.428721:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763792.428769:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff8eb64514bb80 x1876699494181504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763792.428787:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494181504/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763792.428794:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763792.428797:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763792.428800:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494181504:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763792.428819:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181632:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763792.428836:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763792.428845:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff8eb645148000 x1876699494181632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763792.428852:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494181632/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763792.428869:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763792.428871:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763792.428873:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181632:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763792.428877:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181760:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763792.428887:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763792.428892:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff8eb64514b100 x1876699494181760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763792.428904:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494181760/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763792.428909:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763792.428910:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763792.428912:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494181760:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763792.428915:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494181888:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763792.428925:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763792.428931:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763792/real 1789763792] req@ffff8eb645149880 x1876699494181888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763792.428938:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494181888/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763792.428943:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763792.428944:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763792.428946:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494181888:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763793.132551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763794.156520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763795.180562:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763796.204543:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763797.228499:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763797.548453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763797.548463:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763797.548467:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763797.548477:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3802 00000100:00080000:3.0:1789763797.548487:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763797.548494:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763797.548527:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494182016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763797.548553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763797.548557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763797.548559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763797.548563:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3802 00000100:00080000:3.0:1789763797.548568:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763797.548573:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763797.548598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763797.548603:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494182016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763797.548617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494182144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763797.548631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763797.548636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763797.548639:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763797.548643:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3802 00000100:00080000:3.0:1789763797.548649:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763797.548654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789763797.548660:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763797.548664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763797.548680:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494182272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763797.548693:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff8eb645148700 x1876699494182016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763797.548694:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763797.548704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763797.548708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763797.548718:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3802 00000100:00080000:3.0:1789763797.548723:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763797.548727:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494182016/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763797.548732:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00080000:0.0:1789763797.548743:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789763797.548744:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763797.548749:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763797.548753:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494182016:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763797.548761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494182400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763797.548773:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182144:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763797.548794:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763797.548803:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff8eb645149180 x1876699494182144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763797.548811:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494182144/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763797.548817:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763797.548819:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763797.548821:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182144:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763797.548824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182272:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763797.548835:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763797.548841:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff8eb64514ad80 x1876699494182272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763797.548848:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494182272/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763797.548858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763797.548860:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763797.548862:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182272:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763797.548871:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494182400:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763797.548881:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763797.548887:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763797/real 1789763797] req@ffff8eb64514b480 x1876699494182400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763797.548893:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494182400/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763797.548898:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763797.548900:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763797.548903:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494182400:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763798.252528:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763799.276515:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763800.300543:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763801.324499:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763802.348587:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763802.668461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.668475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763802.668481:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.668495:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3807 00000100:00080000:3.0:1789763802.668509:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.668520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763802.668584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494182528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.668614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.668617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.668618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.668622:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3807 00000100:00080000:3.0:1789763802.668625:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.668629:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763802.668647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763802.668664:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494182656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763802.668675:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494182528:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763802.668680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.668684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.668686:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.668689:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3807 00000100:00080000:3.0:1789763802.668692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.668696:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763802.668701:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763802.668711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494182784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763802.668720:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763802.668723:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763802.668724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763802.668727:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3807 00000100:00080000:3.0:1789763802.668730:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763802.668733:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763802.668736:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763802.668744:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494182912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763802.668802:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763802.668834:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763802.668853:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494182912:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763803.372580:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763804.396612:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763805.420535:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763805.740693:0:2357:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763805.740729:0:2357:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763805.740742:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763805.740772:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763805.740780:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763805.740790:0:2357:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789763805.740815:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763805] req@ffff8eb64514ad80 x1876699494182656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.740842:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494182656/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.740853:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.740858:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763805.740863:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182656:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763805.740870:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763805] req@ffff8eb645149180 x1876699494182784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.740881:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494182784/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.740890:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.740894:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763805.740897:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494182784:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763805.740910:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763805] req@ffff8eb645148700 x1876699494182912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.740924:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494182912/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.740933:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.740937:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763805.740940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494182912:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763805.740947:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763802/real 1789763805] req@ffff8eb64514b480 x1876699494182528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.740961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494182528/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.740969:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.740972:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763805.740975:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494182528:192.168.203.160@tcp:250:kworker.0 00010000:00010000:0.0:1789763806.444555:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763807.468554:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763807.788494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.788517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763807.788522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.788533:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3812 00000100:00080000:0.0:1789763807.788543:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.788551:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763807.788585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494183040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.788626:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.788633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.788636:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.788640:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3812 00000100:00080000:0.0:1789763807.788645:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.788650:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763807.788660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763807.788671:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494183040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763807.788678:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494183168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.788692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763807.788696:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.788698:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.788703:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3812 00000100:00080000:0.0:1789763807.788718:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.788724:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763807.788729:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763807.788733:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763807.788751:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494183296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763807.788764:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789763807.788769:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff8eb64304ed80 x1876699494183040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763807.788770:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763807.788774:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763807.788784:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3812 00000100:00080000:0.0:1789763807.788792:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763807.788798:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763807.788804:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494183040/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763807.788805:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789763807.788823:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789763807.788825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494183424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763807.788834:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763807.788843:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494183040:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763807.788876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183168:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.788900:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763807.788913:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff8eb64304c000 x1876699494183168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.788925:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494183168/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.788934:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.788937:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763807.788942:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183168:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763807.788948:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183296:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763807.788967:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763807.788979:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff8eb64304d180 x1876699494183296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.788991:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494183296/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.789000:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.789003:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763807.789007:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183296:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763807.789012:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494183424:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763807.789032:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763807.789043:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763807/real 1789763807] req@ffff8eb64304e300 x1876699494183424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763807.789054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494183424/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763807.789061:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763807.789064:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763807.789067:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494183424:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763808.492527:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763809.516542:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763810.540623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763811.564513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763812.588679:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763812.908465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763812.908486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763812.908489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763812.908498:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3817 00000100:00080000:0.0:1789763812.908506:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763812.908524:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763812.908554:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494183552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763812.908582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763812.908585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763812.908586:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763812.908589:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3817 00000100:00080000:0.0:1789763812.908591:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763812.908596:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763812.908604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763812.908612:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494183680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763812.908620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763812.908623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763812.908624:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763812.908626:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3817 00000100:00080000:0.0:1789763812.908628:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763812.908631:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763812.908634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763812.908637:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494183552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763812.908654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494183808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763812.908662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763812.908665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763812.908666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763812.908669:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3817 00000100:00080000:0.0:1789763812.908671:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763812.908673:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763812.908676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763812.908683:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494183936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763812.908710:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763812.908746:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763812/real 1789763812] req@ffff8eb64304e300 x1876699494183552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763812.908761:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494183552/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763812.908769:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763812.908772:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763812.908775:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494183552:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763812.908780:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183680:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763812.908796:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763812.908805:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763812/real 1789763812] req@ffff8eb64304d180 x1876699494183680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763812.908813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494183680/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763812.908819:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763812.908821:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763812.908823:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183680:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763812.908841:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183808:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763812.908852:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763812.908858:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763812/real 1789763812] req@ffff8eb64304c000 x1876699494183808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763812.908871:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494183808/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763812.908876:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763812.908878:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763812.908879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494183808:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763812.908883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494183936:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763812.908892:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763812.908898:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763812/real 1789763812] req@ffff8eb64304ed80 x1876699494183936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763812.908905:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494183936/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763812.908910:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763812.908912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763812.908913:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494183936:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763813.612629:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763814.636600:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763815.660547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763816.684667:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763817.708630:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763818.028611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763818.028624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763818.028629:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763818.028642:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3822 00000100:00080000:0.0:1789763818.028656:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763818.028665:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763818.028717:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494184064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763818.028753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763818.028757:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763818.028760:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763818.028765:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3822 00000100:00080000:0.0:1789763818.028768:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763818.028771:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763818.028784:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763818.028795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494184192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763818.028804:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763818.028808:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763818.028809:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763818.028812:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3822 00000100:00080000:0.0:1789763818.028814:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763818.028817:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763818.028821:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763818.028848:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494184320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763818.028848:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494184064:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763818.028867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763818.028870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763818.028871:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763818.028875:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3822 00000100:00080000:0.0:1789763818.028878:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763818.028880:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763818.028884:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763818.028895:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494184448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763818.028969:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763818.029031:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff8eb64304c380 x1876699494184064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763818.029058:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494184064/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763818.029073:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763818.029078:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763818.029085:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494184064:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763818.029094:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184192:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763818.029133:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763818.029150:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff8eb64304dc00 x1876699494184192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763818.029166:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494184192/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763818.029176:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763818.029181:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763818.029185:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184192:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763818.029231:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184320:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763818.029257:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763818.029272:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff8eb64304ce00 x1876699494184320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763818.029287:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494184320/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763818.029298:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763818.029302:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763818.029306:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184320:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763818.029315:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494184448:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763818.029340:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763818.029355:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763818/real 1789763818] req@ffff8eb64304f800 x1876699494184448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763818.029369:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494184448/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763818.029379:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763818.029383:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763818.029387:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494184448:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763818.732648:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763819.756587:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763820.780566:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763821.804577:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763822.828499:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763823.148502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.148513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763823.148518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.148529:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3827 00000100:00080000:3.0:1789763823.148553:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.148560:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763823.148593:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494184576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.148614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.148617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.148619:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.148622:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3827 00000100:00080000:3.0:1789763823.148626:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.148628:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763823.148636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763823.148644:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494184576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:3.0:1789763823.148648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494184704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.148659:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.148662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.148663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.148666:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3827 00000100:00080000:3.0:1789763823.148669:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.148671:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763823.148674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763823.148682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494184832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763823.148689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763823.148692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763823.148694:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763823.148696:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3827 00000100:00080000:3.0:1789763823.148698:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763823.148702:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763823.148707:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763823.148707:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763823.148718:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494184960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763823.148757:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763823] req@ffff8eb645149880 x1876699494184576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763823.148774:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494184576/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763823.148782:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763823.148785:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763823.148788:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494184576:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763823.148806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184704:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763823.148826:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763823.148835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763823] req@ffff8eb64514b100 x1876699494184704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763823.148849:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494184704/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763823.148855:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763823.148857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763823.148860:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184704:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763823.148863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184832:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763823.148876:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763823.148882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763823] req@ffff8eb645148000 x1876699494184832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763823.148890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494184832/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763823.148895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763823.148897:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763823.148899:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494184832:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763823.148902:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494184960:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763823.148912:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763823.148917:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763823/real 1789763823] req@ffff8eb64514bb80 x1876699494184960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763823.148929:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494184960/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763823.148934:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763823.148936:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763823.148938:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494184960:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763823.852557:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763824.876559:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763825.900538:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763826.924551:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763827.948510:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763828.268481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763828.268490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763828.268494:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763828.268504:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3832 00000100:00080000:3.0:1789763828.268514:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763828.268521:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763828.268556:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494185088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763828.268582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763828.268585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763828.268587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763828.268590:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3832 00000100:00080000:3.0:1789763828.268593:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763828.268596:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763828.268604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763828.268615:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494185216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763828.268626:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763828.268629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763828.268631:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763828.268634:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3832 00000100:00080000:3.0:1789763828.268637:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763828.268640:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763828.268643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763828.268651:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494185344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763828.268662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763828.268669:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763828.268671:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763828.268672:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494185088:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763828.268677:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3832 00000100:00080000:3.0:1789763828.268688:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763828.268693:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763828.268697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763828.268717:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494185472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763828.268770:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763828.268821:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff8eb64514b800 x1876699494185088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.268844:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494185088/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.268858:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.268863:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763828.268868:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494185088:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763828.268876:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185216:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763828.268897:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763828.268907:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff8eb645148e00 x1876699494185216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.268915:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494185216/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.268935:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.268938:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763828.268940:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185216:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763828.268966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185344:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763828.268981:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763828.268990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff8eb645148380 x1876699494185344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.269012:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494185344/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.269022:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.269026:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763828.269030:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185344:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763828.269038:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494185472:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763828.269059:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763828.269079:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763828/real 1789763828] req@ffff8eb64514a300 x1876699494185472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763828.269088:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494185472/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763828.269093:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763828.269096:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763828.269099:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494185472:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763828.972590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763829.996559:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763831.020552:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763832.044544:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763833.068608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763833.388484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.388492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763833.388496:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.388503:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3837 00000100:00080000:0.0:1789763833.388512:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.388518:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763833.388547:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494185600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.388570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.388573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763833.388574:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.388577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3837 00000100:00080000:0.0:1789763833.388579:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.388581:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763833.388590:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763833.388598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494185728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.388606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.388609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763833.388611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.388618:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3837 00000100:00100000:1.0:1789763833.388619:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494185600:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763833.388622:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.388629:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763833.388635:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763833.388649:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494185856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763833.388662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763833.388667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763833.388668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763833.388671:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3837 00000100:00080000:0.0:1789763833.388675:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763833.388680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763833.388682:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763833.388687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763833.388699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494185984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763833.388725:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff8eb64304d500 x1876699494185600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.388741:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494185600/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.388749:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.388753:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763833.388756:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494185600:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763833.388761:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185728:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.388775:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763833.388792:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff8eb64304d880 x1876699494185728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.388800:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494185728/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.388805:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.388807:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763833.388810:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185728:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763833.388827:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185856:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763833.388840:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763833.388847:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff8eb64304f480 x1876699494185856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.388855:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494185856/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.388860:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.388862:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763833.388864:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494185856:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763833.388868:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494185984:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763833.388877:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763833.388882:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763833/real 1789763833] req@ffff8eb64304e680 x1876699494185984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763833.388890:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494185984/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763833.388895:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763833.388896:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763833.388898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494185984:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763834.092512:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763835.116609:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763836.140586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763837.164723:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763838.188611:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763838.508555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763838.508572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763838.508579:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763838.508594:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3843 00000100:00080000:0.0:1789763838.508622:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763838.508633:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763838.508695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494186112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763838.508739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763838.508744:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763838.508747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763838.508752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3843 00000100:00080000:0.0:1789763838.508756:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763838.508761:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763838.508775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763838.508813:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494186240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763838.508828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763838.508833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763838.508836:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763838.508841:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3843 00000100:00080000:0.0:1789763838.508846:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763838.508850:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763838.508858:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763838.508870:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494186368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763838.508883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494186112:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763838.508889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763838.508914:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763838.508927:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763838.508933:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3843 00000100:00080000:0.0:1789763838.508935:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763838.508938:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763838.508942:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763838.508953:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494186496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763838.509014:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763838.509072:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff8eb64304c700 x1876699494186112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763838.509095:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494186112/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763838.509108:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763838.509112:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763838.509116:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494186112:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763838.509123:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186240:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763838.509144:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763838.509155:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff8eb64304f100 x1876699494186240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763838.509168:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494186240/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763838.509179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763838.509183:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763838.509188:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186240:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763838.509224:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186368:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763838.509254:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763838.509269:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff8eb64304ca80 x1876699494186368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763838.509306:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494186368/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763838.509327:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763838.509331:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763838.509335:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186368:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763838.509341:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494186496:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763838.509363:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763838.509378:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763838/real 1789763838] req@ffff8eb64304ea00 x1876699494186496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763838.509459:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494186496/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763838.509468:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763838.509471:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763838.509474:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494186496:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763839.212593:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763840.236535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763841.260631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763842.284536:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763843.308622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763843.628506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763843.628532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763843.628538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763843.628552:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3848 00000100:00080000:0.0:1789763843.628566:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763843.628591:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763843.628639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494186624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763843.628672:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763843.628677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763843.628680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763843.628684:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3848 00000100:00080000:0.0:1789763843.628689:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763843.628693:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763843.628705:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763843.628729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494186752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763843.628755:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763843.628760:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763843.628763:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763843.628772:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494186624:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763843.628773:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3848 00000100:00080000:0.0:1789763843.628781:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763843.628792:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763843.628799:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763843.628815:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494186880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763843.628828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763843.628833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763843.628835:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763843.628841:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3848 00000100:00080000:0.0:1789763843.628867:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763843.628876:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763843.628885:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763843.628901:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763843.628950:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494187008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763843.628991:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff8eb64304ea00 x1876699494186624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763843.629023:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494186624/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763843.629038:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763843.629044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763843.629050:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494186624:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763843.629081:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186752:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763843.629112:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763843.629128:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff8eb64304ca80 x1876699494186752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763843.629144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494186752/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763843.629155:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763843.629159:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763843.629195:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186752:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763843.629204:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186880:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763843.629229:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763843.629262:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff8eb64304f100 x1876699494186880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763843.629279:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494186880/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763843.629295:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763843.629299:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763843.629303:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494186880:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763843.629317:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494187008:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763843.629343:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763843.629357:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763843/real 1789763843] req@ffff8eb64304c700 x1876699494187008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763843.629373:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494187008/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763843.629389:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763843.629464:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763843.629473:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494187008:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763844.332573:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763845.356550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763846.380502:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763847.404540:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763848.428537:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763848.748472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763848.748482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763848.748485:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763848.748493:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3853 00000100:00080000:3.0:1789763848.748509:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763848.748514:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763848.748566:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494187136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763848.748586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763848.748589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763848.748590:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763848.748610:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3853 00000100:00080000:3.0:1789763848.748613:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763848.748615:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763848.748623:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763848.748632:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494187264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763848.748641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763848.748644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763848.748646:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763848.748649:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3853 00000100:00080000:3.0:1789763848.748653:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763848.748655:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494187136:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763848.748658:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763848.748666:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763848.748674:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494187392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763848.748681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763848.748692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763848.748695:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763848.748700:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3853 00000100:00080000:3.0:1789763848.748707:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763848.748711:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763848.748716:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:3.0:1789763848.748721:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763848.748739:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494187520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763848.748753:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff8eb645149500 x1876699494187136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763848.748776:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494187136/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763848.748787:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763848.748792:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763848.748796:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494187136:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763848.748804:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187264:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763848.748829:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763848.748843:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff8eb645148a80 x1876699494187264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763848.748864:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494187264/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763848.748873:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763848.748877:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763848.748881:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187264:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763848.748904:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187392:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763848.748923:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763848.748932:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff8eb64514a680 x1876699494187392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763848.748949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494187392/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763848.748955:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763848.748957:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763848.748959:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187392:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763848.748963:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494187520:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763848.748975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763848.748981:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763848/real 1789763848] req@ffff8eb645149c00 x1876699494187520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763848.748989:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494187520/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763848.748998:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763848.749001:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763848.749003:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494187520:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763849.452592:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763850.476631:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763851.500556:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763852.524635:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763853.548636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763853.868504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763853.868515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763853.868520:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763853.868532:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3858 00000100:00080000:3.0:1789763853.868545:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763853.868554:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763853.868621:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149c00 x1876699494187648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763853.868660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763853.868666:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763853.868668:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763853.868681:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3858 00000100:00080000:3.0:1789763853.868686:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763853.868691:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763853.868703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763853.868719:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a680 x1876699494187776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763853.868733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763853.868738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763853.868741:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763853.868745:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3858 00000100:00080000:3.0:1789763853.868748:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763853.868757:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763853.868762:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494187648:192.168.203.160@tcp:250:kworker.0 00000008:00100000:3.0:1789763853.868765:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763853.868803:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148a80 x1876699494187904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763853.868817:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763853.868822:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763853.868824:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763853.868829:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3858 00000100:00080000:3.0:1789763853.868836:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763853.868841:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763853.868848:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763853.868860:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149500 x1876699494188032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763853.868874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763853.868944:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763853/real 1789763853] req@ffff8eb645149c00 x1876699494187648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763853.868970:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149c00 x1876699494187648/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763853.868984:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763853.868989:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763853.868994:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494187648:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763853.869004:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187776:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763853.869030:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763853.869045:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763853/real 1789763853] req@ffff8eb64514a680 x1876699494187776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763853.869060:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a680 x1876699494187776/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763853.869070:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763853.869075:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763853.869079:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187776:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763853.869106:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187904:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763853.869127:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763853.869138:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763853/real 1789763853] req@ffff8eb645148a80 x1876699494187904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763853.869152:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148a80 x1876699494187904/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763853.869162:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763853.869213:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763853.869219:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494187904:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763853.869227:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494188032:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763853.869253:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763853.869266:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763853/real 1789763853] req@ffff8eb645149500 x1876699494188032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763853.869280:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149500 x1876699494188032/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763853.869289:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763853.869293:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763853.869297:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494188032:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763854.572726:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763855.596669:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763856.620603:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763857.644623:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763858.668562:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763858.988541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763858.988549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763858.988552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763858.988561:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3863 00000100:00080000:0.0:1789763858.988570:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763858.988575:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763858.988629:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494188160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763858.988668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763858.988672:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763858.988674:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763858.988678:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3863 00000100:00080000:0.0:1789763858.988681:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763858.988687:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763858.988697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763858.988711:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494188288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763858.988720:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494188160:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763858.988726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763858.988731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763858.988732:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763858.988746:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3863 00000100:00080000:0.0:1789763858.988749:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763858.988752:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763858.988757:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763858.988768:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494188416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763858.988783:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763858.988788:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763858.988791:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763858.988796:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3863 00000100:00080000:0.0:1789763858.988802:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763858.988808:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763858.988816:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763858.988820:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763858.988840:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494188544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763858.988881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763858/real 1789763858] req@ffff8eb64304e680 x1876699494188160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763858.988902:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494188160/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763858.988942:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763858.988945:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763858.988948:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494188160:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763858.988971:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188288:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763858.989012:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763858.989025:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763858/real 1789763858] req@ffff8eb64304f480 x1876699494188288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763858.989038:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494188288/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763858.989046:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763858.989049:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763858.989052:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188288:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763858.989058:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188416:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763858.989077:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763858.989092:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763858/real 1789763858] req@ffff8eb64304d880 x1876699494188416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763858.989147:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494188416/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763858.989167:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763858.989170:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763858.989174:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188416:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763858.989180:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494188544:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763858.989222:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763858.989239:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763858/real 1789763858] req@ffff8eb64304d500 x1876699494188544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763858.989251:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494188544/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763858.989258:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763858.989261:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763858.989264:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494188544:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763859.692662:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763860.716652:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763861.740544:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763862.764659:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763863.788646:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763864.108502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763864.108513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763864.108518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763864.108529:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3868 00000100:00080000:3.0:1789763864.108541:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763864.108548:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763864.108600:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514a300 x1876699494188672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763864.108627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763864.108631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763864.108633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763864.108637:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3868 00000100:00080000:3.0:1789763864.108640:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763864.108644:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763864.108653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763864.108665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148380 x1876699494188800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763864.108678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763864.108682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763864.108684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763864.108690:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494188672:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763864.108691:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3868 00000100:00080000:3.0:1789763864.108697:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763864.108703:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763864.108708:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763864.108721:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148e00 x1876699494188928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763864.108732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763864.108735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763864.108737:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763864.108740:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3868 00000100:00080000:3.0:1789763864.108744:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763864.108747:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763864.108751:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763864.108761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b800 x1876699494189056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763864.108770:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763864.108817:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff8eb64514a300 x1876699494188672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.108836:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514a300 x1876699494188672/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.108847:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.108851:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763864.108855:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494188672:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763864.108877:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188800:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763864.108906:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763864.108924:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff8eb645148380 x1876699494188800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.108961:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148380 x1876699494188800/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.108976:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.108981:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763864.108985:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188800:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763864.108993:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188928:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763864.109037:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763864.109051:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff8eb645148e00 x1876699494188928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.109065:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148e00 x1876699494188928/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.109075:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.109079:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763864.109082:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494188928:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763864.109119:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494189056:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763864.109163:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763864.109175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763864/real 1789763864] req@ffff8eb64514b800 x1876699494189056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763864.109186:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b800 x1876699494189056/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763864.109193:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763864.109196:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763864.109199:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494189056:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763864.812503:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763865.836553:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763866.860546:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763867.884561:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763868.908593:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763869.228507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763869.228521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763869.228526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763869.228540:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3873 00000100:00080000:3.0:1789763869.228555:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763869.228564:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763869.228614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514bb80 x1876699494189184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763869.228651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763869.228656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763869.228658:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763869.228663:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3873 00000100:00080000:3.0:1789763869.228667:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763869.228672:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763869.228684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763869.228699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148000 x1876699494189312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763869.228711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763869.228716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763869.228718:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763869.228721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3873 00000100:00080000:3.0:1789763869.228725:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763869.228746:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763869.228753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763869.228766:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b100 x1876699494189440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763869.228780:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763869.228784:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763869.228786:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763869.228812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494189184:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763869.228812:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3873 00000100:00080000:3.0:1789763869.228823:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763869.228834:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763869.228840:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763869.228857:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149880 x1876699494189568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763869.228997:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189312:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763869.229117:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763869.229152:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494189568:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763869.932531:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763870.956638:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763871.980485:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763872.300597:0:2354:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:0.0:1789763872.300634:0:2354:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763872.300647:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763872.300679:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763872.300688:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:0.0:1789763872.300697:0:2354:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:1.0:1789763872.300728:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763872] req@ffff8eb64514b100 x1876699494189440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763872.300750:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b100 x1876699494189440/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763872.300759:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763872.300763:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763872.300766:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189440:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763872.300772:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763872] req@ffff8eb645149880 x1876699494189568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763872.300780:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149880 x1876699494189568/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763872.300786:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763872.300790:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763872.300793:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494189568:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763872.300800:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763872] req@ffff8eb64514bb80 x1876699494189184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763872.300813:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514bb80 x1876699494189184/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763872.300821:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763872.300824:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763872.300828:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494189184:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763872.300834:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763869/real 1789763872] req@ffff8eb645148000 x1876699494189312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763872.300845:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148000 x1876699494189312/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763872.300854:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763872.300858:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763872.300869:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189312:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763873.004519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763874.028570:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763874.348495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763874.348507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763874.348511:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763874.348523:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3878 00000100:00080000:3.0:1789763874.348536:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763874.348544:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763874.348585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494189696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763874.348639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763874.348645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763874.348647:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763874.348651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3878 00000100:00080000:3.0:1789763874.348655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763874.348660:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763874.348670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763874.348685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494189824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763874.348691:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494189696:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763874.348707:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763874.348711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763874.348713:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763874.348716:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3878 00000100:00080000:3.0:1789763874.348721:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763874.348724:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763874.348729:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763874.348739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494189952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763874.348746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763874.348749:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763874.348750:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763874.348752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3878 00000100:00080000:3.0:1789763874.348755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763874.348759:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763874.348763:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763874.348768:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763874.348777:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494190080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763874.348804:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff8eb64514b480 x1876699494189696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.348825:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494189696/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.348836:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.348841:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763874.348845:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494189696:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763874.348895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189824:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763874.348944:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763874.348968:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff8eb645148700 x1876699494189824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.348981:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494189824/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.348990:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.348993:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763874.348996:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189824:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763874.349002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189952:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763874.349050:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763874.349063:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff8eb645149180 x1876699494189952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.349076:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494189952/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.349085:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.349088:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763874.349091:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494189952:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763874.349098:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494190080:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763874.349127:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763874.349138:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763874/real 1789763874] req@ffff8eb64514ad80 x1876699494190080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763874.349153:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494190080/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763874.349162:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763874.349165:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763874.349169:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494190080:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763875.052501:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763876.076586:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763877.100547:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763878.124671:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763879.148634:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763879.468593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.468606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763879.468612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.468624:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3883 00000100:00080000:0.0:1789763879.468636:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.468646:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763879.468690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494190208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.468728:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.468735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.468740:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.468748:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3883 00000100:00080000:0.0:1789763879.468755:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.468764:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763879.468779:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.468808:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494190336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.468830:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.468835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.468842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789763879.468844:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494190208:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763879.468852:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3883 00000100:00080000:0.0:1789763879.468861:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763879.468867:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763879.468876:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763879.468897:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494190464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763879.468912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763879.468917:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763879.468920:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763879.468926:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3883 00000100:00080000:0.0:1789763879.468931:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763879.468940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763879.468942:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763879.468969:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763879.469010:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494190592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763879.469028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff8eb64304f800 x1876699494190208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.469069:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494190208/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.469084:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.469089:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763879.469094:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494190208:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763879.469124:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190336:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.469152:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763879.469168:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff8eb64304ce00 x1876699494190336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.469183:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494190336/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.469193:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.469197:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763879.469201:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190336:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763879.469208:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190464:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763879.469227:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763879.469244:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff8eb64304dc00 x1876699494190464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.469258:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494190464/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.469276:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.469279:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763879.469283:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190464:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763879.469289:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494190592:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763879.469307:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763879.469318:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763879/real 1789763879] req@ffff8eb64304c380 x1876699494190592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763879.469331:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494190592/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763879.469341:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763879.469344:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763879.469348:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494190592:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763880.172669:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763881.196587:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763882.220608:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763883.244709:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763884.268689:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763884.588536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.588550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763884.588556:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.588572:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3889 00000100:00080000:0.0:1789763884.588589:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.588601:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763884.588658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494190720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.588703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.588708:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.588711:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.588717:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3889 00000100:00080000:0.0:1789763884.588722:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.588728:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763884.588740:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763884.588758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494190848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.588771:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.588775:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.588778:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.588782:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3889 00000100:00080000:0.0:1789763884.588786:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763884.588792:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763884.588798:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763884.588809:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494190976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763884.588822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763884.588827:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763884.588829:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763884.588835:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3889 00000100:00080000:0.0:1789763884.588839:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763884.588846:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494190720:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763884.588848:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763884.588860:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763884.588877:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494191104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763884.588975:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763884.589028:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff8eb64304ed80 x1876699494190720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.589054:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494190720/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.589068:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.589074:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763884.589080:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494190720:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763884.589090:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190848:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763884.589119:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763884.589133:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff8eb64304c000 x1876699494190848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.589148:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494190848/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.589159:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.589163:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763884.589167:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190848:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763884.589176:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190976:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763884.589211:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763884.589232:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff8eb64304d180 x1876699494190976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.589257:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494190976/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.589275:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.589281:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763884.589288:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494190976:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763884.589327:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494191104:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763884.589358:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763884.589373:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763884/real 1789763884] req@ffff8eb64304e300 x1876699494191104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763884.589388:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494191104/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763884.589443:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763884.589448:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763884.589453:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494191104:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763885.292654:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763886.316646:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763887.340637:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763888.364642:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763889.388683:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763889.708589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763889.708602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763889.708609:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763889.708623:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3894 00000100:00080000:1.0:1789763889.708637:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763889.708646:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763889.708694:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494191232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763889.708740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763889.708744:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763889.708747:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763889.708752:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3894 00000100:00080000:1.0:1789763889.708758:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763889.708763:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763889.708778:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763889.708794:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494191360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763889.708809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763889.708820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763889.708824:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494191232:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763889.708825:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763889.708852:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3894 00000100:00080000:1.0:1789763889.708856:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763889.708861:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763889.708868:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763889.708887:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494191488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763889.708900:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763889.708928:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763889.708930:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763889.708937:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3894 00000100:00080000:1.0:1789763889.708942:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763889.708949:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763889.708960:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763889.708976:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494191616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763889.708999:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763889.709080:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763889] req@ffff8eb65016c700 x1876699494191232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763889.709121:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494191232/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763889.709140:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763889.709149:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763889.709157:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494191232:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763889.709208:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191360:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763889.709253:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763889.709273:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763889] req@ffff8eb65016ce00 x1876699494191360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763889.709288:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494191360/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763889.709299:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763889.709304:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763889.709308:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191360:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763889.709316:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191488:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763889.709349:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763889.709369:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763889] req@ffff8eb65016e300 x1876699494191488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763889.709433:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494191488/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763889.709455:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763889.709460:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763889.709465:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191488:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763889.709474:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494191616:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763889.709506:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763889.709526:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763889/real 1789763889] req@ffff8eb65016c380 x1876699494191616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763889.709547:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494191616/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763889.709563:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763889.709570:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763889.709577:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494191616:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763890.412525:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763891.436648:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763892.460575:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763893.484676:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763894.508654:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763894.828549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763894.828561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763894.828567:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763894.828578:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3899 00000100:00080000:3.0:1789763894.828591:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763894.828600:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:3.0:1789763894.828653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514ad80 x1876699494191744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763894.828719:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763894.828724:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763894.828727:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763894.828733:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3899 00000100:00080000:3.0:1789763894.828739:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763894.828747:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763894.828760:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763894.828782:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645149180 x1876699494191872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763894.828793:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494191744:192.168.203.160@tcp:250:kworker.0 00000100:00080000:3.0:1789763894.828820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763894.828825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763894.828829:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763894.828835:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3899 00000100:00080000:3.0:1789763894.828839:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763894.828845:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763894.828852:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763894.828918:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb645148700 x1876699494192000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763894.828934:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:3.0:1789763894.828935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763894.828944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763894.828947:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763894.828955:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3899 00000100:00080000:3.0:1789763894.828967:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763894.828972:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:3.0:1789763894.828979:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763894.828993:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff8eb64514ad80 x1876699494191744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763894.829007:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64514b480 x1876699494192128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763894.829026:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514ad80 x1876699494191744/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763894.829048:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763894.829054:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763894.829059:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494191744:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763894.829086:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191872:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763894.829113:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763894.829126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff8eb645149180 x1876699494191872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763894.829139:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645149180 x1876699494191872/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763894.829148:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763894.829152:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763894.829156:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494191872:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763894.829162:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192000:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763894.829179:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763894.829188:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff8eb645148700 x1876699494192000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763894.829200:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb645148700 x1876699494192000/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763894.829209:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763894.829212:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763894.829215:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb645148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192000:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763894.829221:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494192128:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763894.829237:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763894.829247:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763894/real 1789763894] req@ffff8eb64514b480 x1876699494192128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763894.829259:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64514b480 x1876699494192128/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763894.829267:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763894.829270:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763894.829273:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494192128:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763895.532633:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763896.556636:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763897.580684:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763898.604600:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763899.628687:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763899.948596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763899.948611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763899.948618:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763899.948633:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3904 00000100:00080000:1.0:1789763899.948651:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763899.948663:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763899.948723:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494192256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763899.948777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763899.948783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763899.948785:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763899.948791:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3904 00000100:00080000:1.0:1789763899.948796:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763899.948800:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763899.948816:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763899.948850:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494192384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763899.948866:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763899.948881:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763899.948884:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763899.948890:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3904 00000100:00080000:1.0:1789763899.948903:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763899.948909:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494192256:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763899.948914:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763899.948928:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763899.948946:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494192512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763899.948962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763899.948967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763899.948970:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763899.948975:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3904 00000100:00080000:1.0:1789763899.949001:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763899.949007:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763899.949024:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763899.949038:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763899.949044:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494192640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763899.949113:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763899/real 1789763899] req@ffff8eb65016fb80 x1876699494192256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763899.949142:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494192256/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763899.949169:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763899.949174:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763899.949180:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494192256:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763899.949190:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192384:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763899.949219:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763899.949245:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763899/real 1789763899] req@ffff8eb65016d500 x1876699494192384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763899.949260:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494192384/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763899.949270:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763899.949274:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763899.949278:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192384:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763899.949316:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192512:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763899.949339:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763899.949351:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763899/real 1789763899] req@ffff8eb65016d880 x1876699494192512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763899.949366:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494192512/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763899.949376:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763899.949379:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763899.949383:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192512:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763899.949389:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494192640:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763899.949491:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763899.949506:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763899/real 1789763899] req@ffff8eb65016ea00 x1876699494192640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763899.949531:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494192640/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763899.949542:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763899.949545:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763899.949549:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494192640:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763900.652577:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763901.676615:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763902.700599:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763903.724654:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763904.748673:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763905.068595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.068617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763905.068624:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.068640:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3909 00000100:00080000:0.0:1789763905.068658:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.068670:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763905.068740:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e300 x1876699494192768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.068821:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.068830:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.068833:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.068841:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3909 00000100:00080000:0.0:1789763905.068848:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.068855:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763905.068883:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763905.068907:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494192768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763905.068910:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d180 x1876699494192896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.068938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.068943:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.068945:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.068952:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3909 00000100:00080000:0.0:1789763905.068956:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.068960:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763905.068968:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763905.068985:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c000 x1876699494193024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763905.068999:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763905.069003:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763905.069006:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763905.069011:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3909 00000100:00080000:0.0:1789763905.069016:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763905.069020:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763905.069025:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763905.069035:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763905.069043:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ed80 x1876699494193152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763905.069127:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff8eb64304e300 x1876699494192768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.069157:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e300 x1876699494192768/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.069173:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.069178:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763905.069184:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494192768:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763905.069225:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192896:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.069268:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763905.069285:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff8eb64304d180 x1876699494192896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.069302:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d180 x1876699494192896/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.069323:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.069328:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763905.069332:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494192896:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763905.069341:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193024:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763905.069366:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763905.069379:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff8eb64304c000 x1876699494193024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.069426:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c000 x1876699494193024/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.069438:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.069442:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763905.069447:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193024:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763905.069454:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494193152:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763905.069480:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763905.069494:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763905/real 1789763905] req@ffff8eb64304ed80 x1876699494193152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763905.069518:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ed80 x1876699494193152/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763905.069528:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763905.069531:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763905.069535:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494193152:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763905.772635:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763906.796642:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763907.820607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763908.844621:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763909.868665:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763910.188573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763910.188587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763910.188592:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763910.188606:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3914 00000100:00080000:0.0:1789763910.188622:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763910.188637:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763910.188686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494193280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763910.188726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763910.188731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763910.188733:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763910.188738:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3914 00000100:00080000:0.0:1789763910.188742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763910.188747:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763910.188780:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763910.188798:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494193408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763910.188812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763910.188817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763910.188820:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763910.188825:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3914 00000100:00080000:0.0:1789763910.188830:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763910.188839:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494193280:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763910.188841:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763910.188859:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763910.188878:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494193536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763910.188893:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763910.188897:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763910.188900:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763910.188906:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3914 00000100:00080000:0.0:1789763910.188911:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763910.188915:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763910.188922:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763910.188935:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494193664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763910.188953:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763910.189024:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff8eb64304c380 x1876699494193280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.189063:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494193280/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.189082:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.189087:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763910.189093:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494193280:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763910.189104:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193408:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763910.189147:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763910.189175:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff8eb64304dc00 x1876699494193408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.189203:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494193408/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.189222:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.189227:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763910.189232:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193408:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763910.189267:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193536:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763910.189299:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763910.189315:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff8eb64304ce00 x1876699494193536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.189347:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494193536/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.189369:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.189376:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763910.189383:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193536:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763910.189470:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494193664:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763910.189506:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763910.189524:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763910/real 1789763910] req@ffff8eb64304f800 x1876699494193664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763910.189542:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494193664/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763910.189560:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763910.189567:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763910.189574:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494193664:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763910.892618:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763911.916590:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763912.940615:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763913.964622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763914.988678:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763915.308538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763915.308550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763915.308556:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763915.308569:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3919 00000100:00080000:0.0:1789763915.308582:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763915.308592:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763915.308638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494193792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763915.308677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763915.308685:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763915.308689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763915.308697:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3919 00000100:00080000:0.0:1789763915.308704:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763915.308712:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763915.308754:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763915.308780:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494193920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763915.308791:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494193792:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763915.308802:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763915.308808:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763915.308811:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763915.308818:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3919 00000100:00080000:0.0:1789763915.308824:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763915.308830:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763915.308839:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763915.308856:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494194048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763915.308872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763915.308879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763915.308882:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763915.308883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:0.0:1789763915.308889:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3919 00000100:00080000:0.0:1789763915.308894:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763915.308900:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763915.308909:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763915.308926:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494194176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763915.308960:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff8eb64304d500 x1876699494193792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763915.308994:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494193792/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.309014:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763915.309024:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763915.309032:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494193792:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763915.309077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193920:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763915.309117:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763915.309135:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff8eb64304d880 x1876699494193920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763915.309151:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494193920/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.309162:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763915.309166:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763915.309170:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494193920:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763915.309177:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194048:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763915.309199:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763915.309211:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff8eb64304f480 x1876699494194048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763915.309225:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494194048/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.309234:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763915.309237:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763915.309241:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194048:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763915.309247:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494194176:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763915.309266:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763915.309277:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763915/real 1789763915] req@ffff8eb64304e680 x1876699494194176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763915.309290:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494194176/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763915.309300:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763915.309303:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763915.309307:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494194176:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763916.012678:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763917.036688:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763918.060579:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763919.084718:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763920.108624:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763920.428564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763920.428580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763920.428587:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763920.428603:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3924 00000100:00080000:1.0:1789763920.428620:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763920.428631:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763920.428765:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494194304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763920.428811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763920.428816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763920.428820:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763920.428826:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3924 00000100:00080000:1.0:1789763920.428832:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763920.428838:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763920.428852:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763920.428865:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494194304:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763920.428877:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494194432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763920.428894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763920.428900:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763920.428903:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763920.428909:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3924 00000100:00080000:1.0:1789763920.428928:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763920.428933:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763920.428941:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763920.428942:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763920.428972:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494194560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763920.428987:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff8eb65016f800 x1876699494194304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763920.429004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763920.429014:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494194304/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763920.429017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763920.429029:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763920.429037:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763920.429056:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3924 00000100:00080000:0.0:1789763920.429066:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:1.0:1789763920.429071:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763920.429077:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494194304:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763920.429087:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763920.429098:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763920.429115:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763920.429124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494194688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763920.429168:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763920.429208:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff8eb65016e680 x1876699494194432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763920.429223:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494194432/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763920.429233:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763920.429238:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763920.429242:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194432:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763920.429250:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194560:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763920.429272:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763920.429286:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff8eb65016ea00 x1876699494194560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763920.429299:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494194560/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763920.429308:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763920.429312:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763920.429316:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194560:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763920.429341:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494194688:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763920.429358:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763920.429366:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763920/real 1789763920] req@ffff8eb65016d880 x1876699494194688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763920.429374:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494194688/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763920.429380:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763920.429382:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763920.429384:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494194688:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763921.132639:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763922.156698:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763923.180560:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763924.204686:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763925.228629:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763925.548566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763925.548576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763925.548580:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763925.548591:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3930 00000100:00080000:2.0:1789763925.548602:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763925.548609:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763925.548681:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81500 x1876699494194816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763925.548712:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763925.548716:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763925.548719:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763925.548726:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3930 00000100:00080000:2.0:1789763925.548732:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763925.548738:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763925.548748:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763925.548762:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80380 x1876699494194944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763925.548772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763925.548778:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494194816:192.168.203.160@tcp:250:kworker.0 00000100:00080000:2.0:1789763925.548778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763925.548782:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763925.548791:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3930 00000100:00080000:2.0:1789763925.548794:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763925.548797:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763925.548801:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763925.548813:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80000 x1876699494195072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763925.548822:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763925.548826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763925.548827:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763925.548830:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3930 00000100:00080000:2.0:1789763925.548836:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763925.548839:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763925.548843:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763925.548846:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:2.0:1789763925.548855:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f80a80 x1876699494195200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763925.548920:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff8eb650f81500 x1876699494194816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763925.548964:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81500 x1876699494194816/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763925.548973:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763925.548977:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763925.548980:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494194816:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763925.549002:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194944:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763925.549022:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763925.549032:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff8eb650f80380 x1876699494194944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763925.549041:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80380 x1876699494194944/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763925.549047:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763925.549057:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763925.549060:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494194944:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763925.549065:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195072:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763925.549079:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763925.549087:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff8eb650f80000 x1876699494195072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763925.549096:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80000 x1876699494195072/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763925.549127:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763925.549129:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763925.549132:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195072:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763925.549137:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494195200:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763925.549154:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763925.549163:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763925/real 1789763925] req@ffff8eb650f80a80 x1876699494195200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763925.549173:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f80a80 x1876699494195200/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763925.549179:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763925.549182:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763925.549184:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494195200:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763926.252685:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763927.276693:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763928.300581:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763929.324573:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763930.348625:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763930.668525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.668541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763930.668548:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.668579:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3935 00000100:00080000:0.0:1789763930.668598:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.668637:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763930.668684:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494195328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.668726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.668731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763930.668734:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.668740:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3935 00000100:00080000:0.0:1789763930.668745:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.668750:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763930.668762:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763930.668778:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494195456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.668793:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.668798:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763930.668801:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.668806:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3935 00000100:00080000:0.0:1789763930.668810:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.668819:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763930.668822:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494195328:192.168.203.160@tcp:250:kworker.0 00000008:00100000:0.0:1789763930.668828:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763930.668845:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494195584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763930.668861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763930.668866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763930.668870:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763930.668875:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3935 00000100:00080000:0.0:1789763930.668895:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763930.668905:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:1.0:1789763930.668914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:0.0:1789763930.668915:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763930.668948:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494195712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763930.668982:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff8eb64304c700 x1876699494195328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763930.669014:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494195328/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763930.669029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763930.669035:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763930.669040:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494195328:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763930.669050:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195456:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763930.669077:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763930.669094:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff8eb64304f100 x1876699494195456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763930.669109:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494195456/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763930.669121:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763930.669125:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763930.669129:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195456:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763930.669158:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195584:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763930.669179:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763930.669191:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff8eb64304ca80 x1876699494195584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763930.669205:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494195584/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763930.669216:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763930.669219:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763930.669223:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195584:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763930.669229:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494195712:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763930.669247:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763930.669258:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763930/real 1789763930] req@ffff8eb64304ea00 x1876699494195712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763930.669272:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494195712/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763930.669282:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763930.669286:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763930.669290:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494195712:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763931.372590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763932.396616:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763933.420666:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763934.444592:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763935.468686:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763935.788622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763935.788635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763935.788640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763935.788653:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3940 00000100:00080000:1.0:1789763935.788666:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763935.788675:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763935.788723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494195840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763935.788768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763935.788773:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763935.788775:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763935.788780:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3940 00000100:00080000:1.0:1789763935.788785:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763935.788790:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763935.788801:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763935.788818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494195968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763935.788833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763935.788839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763935.788842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763935.788846:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3940 00000100:00080000:1.0:1789763935.788850:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763935.788858:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763935.788861:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494195840:192.168.203.160@tcp:250:kworker.0 00000008:00100000:1.0:1789763935.788866:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763935.788886:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494196096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763935.788900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763935.788904:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763935.788907:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763935.788913:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3940 00000100:00080000:1.0:1789763935.788917:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763935.788923:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763935.788929:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763935.788943:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494196224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763935.789007:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195968:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763935.789054:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763935.789085:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494196224:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763936.492684:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763937.516657:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763938.540628:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763938.860704:0:2355:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.160:988 00000400:00000100:2.0:1789763938.860742:0:2355:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.160@tcp at host 192.168.203.160 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763938.860756:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763938.860786:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763938.860797:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000800:00000100:2.0:1789763938.860807:0:2355:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.160@tcp 00000100:00100000:0.0:1789763938.860828:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763938] req@ffff8eb65016c380 x1876699494196096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763938.860853:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494196096/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763938.860865:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763938.860870:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763938.860875:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196096:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763938.860885:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763938] req@ffff8eb65016e300 x1876699494196224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763938.860899:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494196224/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763938.860908:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763938.860912:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763938.860915:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494196224:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763938.860923:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763938] req@ffff8eb65016d500 x1876699494195840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763938.860957:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494195840/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763938.860972:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763938.860979:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763938.860986:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494195840:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763938.860999:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763935/real 1789763938] req@ffff8eb65016fb80 x1876699494195968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763938.861022:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494195968/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763938.861051:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763938.861054:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763938.861058:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494195968:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763939.564659:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763940.588607:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763940.908573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763940.908591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763940.908598:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763940.908617:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3945 00000100:00080000:0.0:1789763940.908635:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763940.908646:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763940.908692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ea00 x1876699494196352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763940.908733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763940.908738:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763940.908740:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763940.908746:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3945 00000100:00080000:0.0:1789763940.908751:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763940.908758:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763940.908771:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763940.908787:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ca80 x1876699494196480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763940.908799:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494196352:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763940.908807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763940.908814:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763940.908817:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763940.908842:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3945 00000100:00080000:0.0:1789763940.908851:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763940.908860:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763940.908874:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789763940.908888:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763940.908904:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f100 x1876699494196608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763940.908928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763940.908935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763940.908938:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763940.908945:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3945 00000100:00080000:0.0:1789763940.908954:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789763940.908959:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763940/real 1789763940] req@ffff8eb64304ea00 x1876699494196352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763940.908969:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763940.908988:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763940.909024:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c700 x1876699494196736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763940.909030:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ea00 x1876699494196352/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763940.909059:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763940.909067:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763940.909080:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494196352:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763940.909133:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196480:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763940.909219:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763940.909248:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763940/real 1789763940] req@ffff8eb64304ca80 x1876699494196480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763940.909274:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ca80 x1876699494196480/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763940.909293:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763940.909300:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763940.909308:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196480:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763940.909320:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196608:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763940.909362:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763940.909385:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763940/real 1789763940] req@ffff8eb64304f100 x1876699494196608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763940.909455:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f100 x1876699494196608/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763940.909486:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763940.909492:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763940.909499:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196608:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763940.909513:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494196736:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763940.909581:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763940.909606:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763940/real 1789763940] req@ffff8eb64304c700 x1876699494196736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763940.909629:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c700 x1876699494196736/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763940.909644:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763940.909649:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763940.909655:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494196736:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763941.612661:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763942.636711:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763943.660631:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763944.684607:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763945.708676:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763946.028591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763946.028604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763946.028609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763946.028621:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3950 00000100:00080000:1.0:1789763946.028635:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763946.028644:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763946.028691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494196864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763946.028739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763946.028746:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763946.028750:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763946.028759:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3950 00000100:00080000:1.0:1789763946.028766:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763946.028773:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763946.028789:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763946.028806:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494196864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763946.028813:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494196992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763946.028831:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763946.028837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763946.028840:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763946.028846:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3950 00000100:00080000:1.0:1789763946.028852:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763946.028858:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763946.028866:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763946.028885:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494197120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763946.028903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763946.028909:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763946.028913:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763946.028914:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763946.028920:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3950 00000100:00080000:1.0:1789763946.028926:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763946.028932:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763946.028941:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763946.028958:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494197248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763946.028972:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff8eb65016ce00 x1876699494196864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.029003:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494196864/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.029023:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.029030:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763946.029038:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494196864:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763946.029077:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196992:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763946.029112:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763946.029129:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff8eb65016c700 x1876699494196992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.029144:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494196992/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.029155:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.029159:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763946.029164:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494196992:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763946.029171:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197120:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763946.029191:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763946.029202:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff8eb65016dc00 x1876699494197120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.029216:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494197120/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.029226:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.029229:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763946.029233:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197120:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763946.029240:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494197248:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763946.029258:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763946.029269:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763946/real 1789763946] req@ffff8eb65016df80 x1876699494197248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763946.029284:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494197248/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763946.029293:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763946.029296:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763946.029300:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494197248:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763946.732601:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763947.756603:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763948.780623:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763949.804693:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763950.828569:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763951.148552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763951.148565:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763951.148570:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763951.148583:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3955 00000100:00080000:1.0:1789763951.148596:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763951.148605:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763951.148657:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494197376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763951.148689:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763951.148694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763951.148696:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763951.148703:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3955 00000100:00080000:1.0:1789763951.148706:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763951.148713:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763951.148725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763951.148739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494197504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763951.148750:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494197376:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763951.148754:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763951.148763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763951.148765:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763951.148769:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3955 00000100:00080000:1.0:1789763951.148772:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763951.148775:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763951.148781:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763951.148793:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494197632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763951.148805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763951.148808:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763951.148811:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763951.148814:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3955 00000100:00080000:1.0:1789763951.148818:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763951.148821:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763951.148826:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763951.148834:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494197760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763951.148858:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763951.148930:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff8eb65016ed80 x1876699494197376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.148947:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494197376/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.148957:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.148961:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763951.148964:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494197376:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763951.148989:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197504:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.149006:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763951.149015:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff8eb65016f480 x1876699494197504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.149023:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494197504/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.149029:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.149032:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763951.149034:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197504:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763951.149037:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197632:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763951.149053:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763951.149064:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff8eb65016d180 x1876699494197632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.149077:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494197632/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.149086:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.149090:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763951.149093:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494197632:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763951.149100:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494197760:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763951.149118:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763951.149126:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763951/real 1789763951] req@ffff8eb65016c000 x1876699494197760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763951.149134:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494197760/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763951.149140:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763951.149142:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763951.149145:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494197760:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763951.852652:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763952.876639:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763953.900619:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763954.924668:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763955.948648:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763956.268576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.268603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763956.268614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.268627:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3960 00000100:00080000:0.0:1789763956.268642:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.268651:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763956.268699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304e680 x1876699494197888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.268738:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.268743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.268747:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.268752:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3960 00000100:00080000:0.0:1789763956.268756:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.268762:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763956.268775:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763956.268792:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f480 x1876699494198016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763956.268812:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494197888:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763956.268814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.268839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.268842:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.268848:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3960 00000100:00080000:0.0:1789763956.268852:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.268859:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763956.268865:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763956.268881:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d880 x1876699494198144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763956.268894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763956.268899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763956.268901:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763956.268906:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3960 00000100:00080000:0.0:1789763956.268911:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763956.268916:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763956.268921:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763956.268937:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304d500 x1876699494198272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763956.268940:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763956.269018:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763956] req@ffff8eb64304e680 x1876699494197888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763956.269076:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304e680 x1876699494197888/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763956.269088:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763956.269093:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763956.269098:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494197888:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763956.269142:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198016:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763956.269177:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763956.269191:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763956] req@ffff8eb64304f480 x1876699494198016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763956.269212:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f480 x1876699494198016/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763956.269220:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763956.269223:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763956.269226:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198016:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763956.269233:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198144:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763956.269253:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763956.269262:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763956] req@ffff8eb64304d880 x1876699494198144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763956.269272:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d880 x1876699494198144/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763956.269285:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763956.269288:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763956.269291:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198144:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763956.269296:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494198272:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763956.269310:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763956.269317:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763956/real 1789763956] req@ffff8eb64304d500 x1876699494198272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763956.269326:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304d500 x1876699494198272/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763956.269333:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763956.269335:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763956.269337:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494198272:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763956.972622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763957.996650:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763959.020591:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763960.044590:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763961.068545:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763961.388616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763961.388632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763961.388650:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763961.388665:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3965 00000100:00080000:1.0:1789763961.388681:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763961.388693:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763961.388761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f100 x1876699494198400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763961.388826:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763961.388834:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763961.388838:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763961.388848:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3965 00000100:00080000:1.0:1789763961.388857:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763961.388868:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763961.388884:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763961.388898:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494198400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763961.388934:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ca80 x1876699494198528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763961.388962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763961.388969:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763961.388974:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763961.388982:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3965 00000100:00080000:1.0:1789763961.389003:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763961.389004:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763961.389021:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763961.389033:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763961.389062:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c000 x1876699494198656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763961.389077:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff8eb65016f100 x1876699494198400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.389097:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763961.389107:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763961.389112:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763961.389117:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f100 x1876699494198400/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763961.389128:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3965 00000100:00080000:0.0:1789763961.389144:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763961.389146:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763961.389156:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00080000:1.0:1789763961.389166:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763961.389180:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763961.389182:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494198400:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763961.389218:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d180 x1876699494198784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763961.389227:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198528:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763961.389278:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763961.389301:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff8eb65016ca80 x1876699494198528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763961.389319:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ca80 x1876699494198528/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763961.389330:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763961.389335:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763961.389339:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198528:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763961.389347:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198656:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763961.389367:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763961.389447:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff8eb65016c000 x1876699494198656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763961.389468:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c000 x1876699494198656/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763961.389479:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763961.389483:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763961.389487:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494198656:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763961.389506:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494198784:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763961.389536:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763961.389561:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763961/real 1789763961] req@ffff8eb65016d180 x1876699494198784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763961.389576:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d180 x1876699494198784/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763961.389589:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763961.389595:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763961.389602:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494198784:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763962.092638:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763963.116643:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763964.140653:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763965.164688:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763966.188602:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763966.508544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763966.508557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763966.508563:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763966.508575:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3971 00000100:00080000:1.0:1789763966.508592:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763966.508604:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763966.508664:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f480 x1876699494198912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763966.508713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763966.508720:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763966.508725:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763966.508733:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3971 00000100:00080000:1.0:1789763966.508742:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763966.508751:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763966.508768:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763966.508793:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ed80 x1876699494199040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763966.508811:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494198912:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763966.508817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763966.508837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763966.508841:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763966.508849:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3971 00000100:00080000:1.0:1789763966.508855:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763966.508862:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763966.508869:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763966.508887:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016df80 x1876699494199168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763966.508900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763966.508906:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763966.508910:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763966.508915:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3971 00000100:00080000:1.0:1789763966.508919:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763966.508924:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763966.508933:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763966.508945:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016dc00 x1876699494199296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763966.508974:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763966.509055:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff8eb65016f480 x1876699494198912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763966.509095:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f480 x1876699494198912/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.509121:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763966.509130:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763966.509136:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494198912:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763966.509180:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199040:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763966.509219:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763966.509237:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff8eb65016ed80 x1876699494199040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763966.509264:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ed80 x1876699494199040/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.509276:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763966.509280:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763966.509285:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199040:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763966.509294:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199168:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763966.509319:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763966.509332:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff8eb65016df80 x1876699494199168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763966.509379:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016df80 x1876699494199168/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.509390:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763966.509442:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763966.509447:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199168:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763966.509457:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494199296:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763966.509504:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763966.509529:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763966/real 1789763966] req@ffff8eb65016dc00 x1876699494199296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763966.509554:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016dc00 x1876699494199296/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763966.509582:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763966.509588:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763966.509595:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494199296:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763967.212660:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763968.236632:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763969.260529:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763970.284547:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763971.308590:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763971.628504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.628513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763971.628516:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.628526:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3976 00000100:00080000:0.0:1789763971.628535:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.628541:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:0.0:1789763971.628572:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304f800 x1876699494199424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763971.628596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.628599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.628600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.628603:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3976 00000100:00080000:0.0:1789763971.628606:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.628609:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763971.628616:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763971.628625:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304ce00 x1876699494199552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763971.628632:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494199424:192.168.203.160@tcp:250:kworker.0 00000100:00080000:0.0:1789763971.628635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.628641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.628642:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.628645:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3976 00000100:00080000:0.0:1789763971.628647:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.628649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763971.628653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763971.628661:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304dc00 x1876699494199680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763971.628668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763971.628670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763971.628671:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763971.628674:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3976 00000100:00080000:0.0:1789763971.628686:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763971.628689:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:0.0:1789763971.628693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763971.628700:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb64304c380 x1876699494199808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763971.628706:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763971.628748:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff8eb64304f800 x1876699494199424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763971.628769:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304f800 x1876699494199424/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763971.628780:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763971.628793:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763971.628798:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494199424:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763971.628823:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199552:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763971.628843:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763971.628853:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff8eb64304ce00 x1876699494199552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763971.628861:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304ce00 x1876699494199552/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763971.628867:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763971.628869:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763971.628879:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199552:192.168.203.160@tcp:38:kworker.0 00000100:00100000:1.0:1789763971.628883:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199680:192.168.203.160@tcp:8:kworker.0 00000800:00000100:1.0:1789763971.628895:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763971.628902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff8eb64304dc00 x1876699494199680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763971.628909:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304dc00 x1876699494199680/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763971.628919:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763971.628921:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763971.628924:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494199680:192.168.203.160@tcp:8:kworker.0 00000100:00100000:1.0:1789763971.628927:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494199808:192.168.203.160@tcp:38:kworker.0 00000800:00000100:1.0:1789763971.628936:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763971.628942:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763971/real 1789763971] req@ffff8eb64304c380 x1876699494199808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763971.628949:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb64304c380 x1876699494199808/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763971.628954:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763971.628955:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:1.0:1789763971.628957:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb64304c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494199808:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763972.332527:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763973.356510:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763974.380550:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763975.404551:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763976.428530:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763976.748478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.748486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763976.748489:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.748497:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3981 00000100:00080000:1.0:1789763976.748506:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.748511:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763976.748540:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c700 x1876699494199936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.748568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.748571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763976.748572:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.748575:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3981 00000100:00080000:1.0:1789763976.748577:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.748580:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763976.748588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763976.748597:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ce00 x1876699494200064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.748605:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.748609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763976.748610:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494199936:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763976.748610:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.748618:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3981 00000100:00080000:1.0:1789763976.748620:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.748622:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763976.748626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763976.748636:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016fb80 x1876699494200192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763976.748643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763976.748646:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763976.748647:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763976.748650:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3981 00000100:00080000:1.0:1789763976.748661:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763976.748664:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763976.748668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763976.748672:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:1.0:1789763976.748680:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d500 x1876699494200320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763976.748723:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff8eb65016c700 x1876699494199936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763976.748742:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c700 x1876699494199936/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763976.748750:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763976.748753:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763976.748756:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494199936:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763976.748788:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200064:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763976.748804:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763976.748812:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff8eb65016ce00 x1876699494200064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763976.748820:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ce00 x1876699494200064/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763976.748825:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763976.748828:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763976.748830:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200064:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763976.748833:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200192:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763976.748844:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763976.748849:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff8eb65016fb80 x1876699494200192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763976.748857:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016fb80 x1876699494200192/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763976.748866:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763976.748868:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763976.748870:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200192:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763976.748874:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494200320:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763976.748883:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763976.748889:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763976/real 1789763976] req@ffff8eb65016d500 x1876699494200320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763976.748896:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d500 x1876699494200320/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763976.748901:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763976.748903:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763976.748905:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494200320:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763977.452519:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763978.476521:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763979.500518:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763980.524513:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763981.548525:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763981.868577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763981.868595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763981.868601:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763981.868631:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3986 00000100:00080000:1.0:1789763981.868643:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763981.868651:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763981.868693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e300 x1876699494200448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763981.868722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763981.868725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763981.868727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763981.868731:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3986 00000100:00080000:1.0:1789763981.868733:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763981.868737:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763981.868747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763981.868758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016c380 x1876699494200576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763981.868773:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494200448:192.168.203.160@tcp:250:kworker.0 00000100:00080000:1.0:1789763981.868774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763981.868789:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763981.868792:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763981.868796:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3986 00000100:00080000:1.0:1789763981.868799:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763981.868803:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763981.868809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763981.868826:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494200704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763981.868834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763981.868838:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763981.868839:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763981.868843:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3986 00000100:00080000:1.0:1789763981.868846:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763981.868848:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00080000:1.0:1789763981.868852:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763981.868859:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763981.868874:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494200832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763981.868902:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763981/real 1789763981] req@ffff8eb65016e300 x1876699494200448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763981.868917:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e300 x1876699494200448/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763981.868927:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763981.868930:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763981.868933:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494200448:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763981.868966:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200576:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763981.868982:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763981.868990:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763981/real 1789763981] req@ffff8eb65016c380 x1876699494200576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763981.868998:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016c380 x1876699494200576/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763981.869004:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763981.869006:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763981.869009:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200576:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763981.869013:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200704:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763981.869023:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763981.869029:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763981/real 1789763981] req@ffff8eb65016d880 x1876699494200704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763981.869036:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494200704/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763981.869042:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763981.869044:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763981.869046:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494200704:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763981.869050:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494200832:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763981.869060:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763981.869066:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763981/real 1789763981] req@ffff8eb65016ea00 x1876699494200832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763981.869073:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494200832/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763981.869078:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763981.869080:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763981.869082:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494200832:192.168.203.160@tcp:38:kworker.0 00010000:00010000:1.0:1789763982.572519:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763983.596505:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763984.620498:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763985.644532:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763986.668499:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763986.988472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763986.988480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763986.988484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763986.988492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3991 00000100:00080000:2.0:1789763986.988499:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763986.988505:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:2.0:1789763986.988535:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83800 x1876699494200960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763986.988561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763986.988564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763986.988565:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763986.988569:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3991 00000100:00080000:2.0:1789763986.988573:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763986.988578:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763986.988588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763986.988601:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494200960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:2.0:1789763986.988605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81f80 x1876699494201088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763986.988621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763986.988625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763986.988627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763986.988629:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3991 00000100:00080000:2.0:1789763986.988633:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763986.988636:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:2.0:1789763986.988640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763986.988650:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f83b80 x1876699494201216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763986.988658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763986.988660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763986.988661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763986.988664:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3991 00000100:00080000:2.0:1789763986.988668:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763986.988672:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000800:00000100:0.0:1789763986.988678:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000008:00100000:2.0:1789763986.988678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763986.988695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb650f81180 x1876699494201344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763986.988728:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763986/real 1789763986] req@ffff8eb650f83800 x1876699494200960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763986.988750:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83800 x1876699494200960/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763986.988760:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763986.988765:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763986.988769:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494200960:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763986.988803:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201088:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763986.988825:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763986.988835:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763986/real 1789763986] req@ffff8eb650f81f80 x1876699494201088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763986.988848:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81f80 x1876699494201088/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763986.988854:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763986.988857:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763986.988859:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201088:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763986.988863:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201216:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763986.988874:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763986.988881:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763986/real 1789763986] req@ffff8eb650f83b80 x1876699494201216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763986.988888:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f83b80 x1876699494201216/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763986.988894:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763986.988896:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763986.988898:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201216:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763986.988901:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494201344:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763986.988910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763986.988916:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763986/real 1789763986] req@ffff8eb650f81180 x1876699494201344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763986.988923:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb650f81180 x1876699494201344/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763986.988932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763986.988934:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763986.988936:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb650f81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494201344:192.168.203.160@tcp:38:kworker.0 00010000:00010000:0.0:1789763987.692520:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763988.716531:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763989.740533:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763990.764535:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763991.788503:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763992.108540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96f72786-7ebd-4df8-8048-c06aee14e87f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763992.108554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763992.108560:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763992.108573:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.160@tcp: connect to NID 192.168.203.160@tcp last attempt 3996 00000100:00080000:1.0:1789763992.108587:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.160@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763992.108597:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.160@tcp: import ffff8eb64314b000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000100:00100000:1.0:1789763992.108657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016e680 x1876699494201472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763992.108684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763992.108686:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763992.108688:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763992.108692:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3996 00000100:00080000:1.0:1789763992.108695:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763992.108698:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb6513ab000: import ffff8eb6462fb000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763992.108707:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763992.108717:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016f800 x1876699494201600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763992.108726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9b240726-48d9-467b-83a8-5b29f6d3cb3b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763992.108728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763992.108730:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763992.108732:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: connect to NID 192.168.203.160@tcp last attempt 3996 00000100:00080000:1.0:1789763992.108734:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763992.108737:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8eb6513ab000: import ffff8eb6450cf800 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763992.108740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763992.108744:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494201472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:1.0:1789763992.108753:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016ea00 x1876699494201728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763992.108766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 78fddf19-6995-4f97-8b3a-c5dc3cb1d098->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763992.108769:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763992.108770:0:8:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763992.108773:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: connect to NID 192.168.203.160@tcp last attempt 3996 00000100:00080000:1.0:1789763992.108775:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763992.108778:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8eb648433000: import ffff8eb6431a3000 using connection 192.168.203.160@tcp/192.168.203.160@tcp 00000008:00100000:1.0:1789763992.108781:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763992.108788:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8eb65016d880 x1876699494201856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763992.108817:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763992.108860:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff8eb65016e680 x1876699494201472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763992.108875:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016e680 x1876699494201472/t0(0) o250->MGC192.168.203.160@tcp@192.168.203.160@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763992.108883:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb64314b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763992.108886:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763992.108889:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96f72786-7ebd-4df8-8048-c06aee14e87f:2366:1876699494201472:192.168.203.160@tcp:250:kworker.0 00000100:00100000:0.0:1789763992.108895:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201600:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763992.108910:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763992.108918:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff8eb65016f800 x1876699494201600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763992.108926:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016f800 x1876699494201600/t0(0) o38->lustre-MDT0000-mdc-ffff8eb6513ab000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763992.108932:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6462fb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763992.108935:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763992.108937:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201600:192.168.203.160@tcp:38:kworker.0 00000100:00100000:0.0:1789763992.108968:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201728:192.168.203.160@tcp:8:kworker.0 00000800:00000100:0.0:1789763992.108991:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763992.109003:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff8eb65016ea00 x1876699494201728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763992.109013:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016ea00 x1876699494201728/t0(0) o8->lustre-OST0000-osc-ffff8eb6513ab000@192.168.203.160@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763992.109019:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6450cf800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763992.109021:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763992.109023:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9b240726-48d9-467b-83a8-5b29f6d3cb3b:2366:1876699494201728:192.168.203.160@tcp:8:kworker.0 00000100:00100000:0.0:1789763992.109028:0:2366:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494201856:192.168.203.160@tcp:38:kworker.0 00000800:00000100:0.0:1789763992.109042:0:2366:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.160@tcp 00000100:00100000:0.0:1789763992.109049:0:2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763992/real 1789763992] req@ffff8eb65016d880 x1876699494201856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763992.109056:0:2366:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8eb65016d880 x1876699494201856/t0(0) o38->lustre-MDT0000-mdc-ffff8eb648433000@192.168.203.160@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763992.109062:0:2366:0:(import.c:65:import_set_state_nolock()) ffff8eb6431a3000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763992.109063:0:2366:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.160@tcp failed (-110) 00000100:00100000:0.0:1789763992.109065:0:2366:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8eb65016d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:78fddf19-6995-4f97-8b3a-c5dc3cb1d098:2366:1876699494201856:192.168.203.160@tcp:38:kworker.0 00010000:00010000:3.0:1789763992.812619:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763993.836622:0:11722:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763994.860513:0:5608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763995.884548:0:5607:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 98239 lines, 98239 kept, 0 dropped, 0 bad.